Systems and methods for generating histology image training datasets for machine learning models

ABSTRACT

A system and method are provided for training and using a machine learning model to analyze hematoxylin and eosin (H&amp;E) slide images, where the machine learning model is trained using a training data set comprising a plurality of unmarked H&amp;E images and a plurality of marked H&amp;E images, each marked H&amp;E image being associated with one unmarked H&amp;E image and each marked H&amp;E image including a location of one or more molecules determined by analyzing a multiplex IHC image having at least two IHC stains, each IHC stain having a unique color and a unique target molecule. Predicted molecules and locations identified with the machine learning model result in an immunotherapy response class being assigned to the H&amp;E slide image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.63/199,185, filed Dec. 11, 2020, which is incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

The present disclosure relates to examining digital images to detect,quantify, and/or characterize cancer-related biomarker(s) and, moreparticularly, to detect, quantify, and/or characterize such biomarkersfrom analysis of one or more histopathology slide images.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Clinical success of immunotherapy cancer treatment (sometimes referredto as immuno-oncology treatment) is driving the need for new prognosticand predictive assays to inform patient selection and stratification.One way to assess potential immunotherapies is by detecting certain celltypes, in a biological specimen, that may be associated with a betterclinical outcome for patients (see, e.g., Chuah et al. Journal forImmunotherapy of cancer, 2019).

Commonly, immunotherapy response predictions are made based on examiningimmunohistochemistry (IHC) stained slides. For example, a thin slice oftumor tissue (approximately 5 microns thick) or a blood smear of cancercells is affixed to glass microscope slides to create a histology slide,also known as a pathology slide, and that slide is submerged in a liquidsolution containing an IHC stain of antibodies. Each antibody isdesigned to bind to one copy of the target biomarker molecule on theslide and is coupled with an enzyme that then converts a substrate intoa visible dye. This stain allows a trained pathologist or other trainedanalyst to visually inspect the location of target molecules on theslide. Similarly, immunofluorescence (IF) is another imaging technique,one that relies upon antibodies chemically labeled with fluorescent dyesto visualize target molecules in a sample, in particular informalin-fixed, paraffin-embedded (FFPE) specimens mounted on glassslides.

However, IHC stained slides and/or manual annotations of IHC stainedslides can be costly and time consuming to generate. In addition, ifonly one molecule is stained in an IHC slide, it is difficult to analyzeinteractions between multiple molecules, which may be indicative of thelikelihood of a tumor to respond to immunotherapy.

Techniques have been proposed for overcoming conventional limitationswith IHC image analysis using trained machine learning models. Howeverthe number of stains that are used is limited and manual imageannotation is still required for ground truth development and modeltraining. There is a need for IHC image-informed machine learning modelscapable of examining histology images for tissue identification andcancer prediction.

SUMMARY OF THE INVENTION

The present application presents systems and methods of using multipleximmunohistochemistry (IHC) slide images and data to train machinelearning models, in particular, to train models capable of predictingfluorescent multiplex immunochemistry (mIHC) information andimmunotherapy response likelihood from histopathology images, namelyHematoxylin and eosin (H&E) slide images. In this way, techniques areprovided for training learning models capable of making informativeimmunotherapy predictions from H&E slide images, where the trainingdataset includes IHC slide images. In use, these trained learning modelsare able to make informative immunotherapy predictions from non-stainedhistopathology slide images, whether H&E slide images, brightfieldimages, or others.

This technique may also be used to characterize subtypes of immune cellsand their functional role in auto-immune diseases (for example, multiplesclerosis, lupus or SLE, type I diabetes, etc.) and inform treatmentresponse as well as disease progression. Additional IHC and/orimmunofluorescence staining (IF) stains may be used to target moleculesthat are not disclosed here, especially immune cell markers.

In accordance with an embodiment, a method for using a machine learningmodel to analyze at least one hematoxylin and eosin slide (H&E) image,the method comprising: a. receiving, at one or more processors, the H&Eslide image; b. using, at the one or more processors, a machine learningmodel to predict locations of molecules in the H&E slide image, wherethe machine learning model is trained using a training data setcomprising a plurality of unmarked H&E images and a plurality of markedH&E images, each marked H&E image being associated with one unmarked H&Eimage and each marked H&E image including a location of one or moremolecules determined by analyzing a multiplex IHC image having at leasttwo IHC stains, wherein each IHC stain has a unique color and a uniquetarget molecule and wherein analyzing the multiplex IHC image includesdetermining an IHC stain that contributes to any two or more overlappingor adjacent IHC stains and comparing each IHC stain in the multiplex IHCimage to a threshold; c. analyzing the number of predicted molecules andlocations of the predicted molecules; and d. assigning an immunotherapyresponse class to the H&E slide image, based on the number of predictedmolecules and/or locations of the predicted molecules.

In some examples, the molecules are immunotherapy biomarkers examples ofwhich include CD3, CD8, CD20, CD68, CK, PD1, and PDL1.

In some examples, the method further comprises locating individualcells.

In some examples, method further comprises inferring, using a machinelearning model, cell types for at least one of the individual cells.

In some examples, method further comprises predicting an immunotherapyresponse of the patient based, at least partially, on the inferred celltypes.

In some examples, method further comprises, for each individual cellassociated with two or more classes of stained molecules, calculatingthe proportions of each stained molecule associated with the individualcell. For example, if a cell has two or more classes of stained molecule(for example, two or more target molecules) associated with it (forexample, the stained molecules appear to be located on and/or in thecell), for each class of stained molecule, a ratio, proportion, orpercentage (for example, proportion or percentage of total stainedmolecules associated with the cell) may be calculated.

In some examples, method further comprises predicting an immunotherapyresponse of the patient based, at least partially, on the calculatedproportions of each stained molecule associated with each individualcell.

In some examples, method further comprises calculating a multifacetedscore based on imaging features and genetic features.

In some examples, method further comprises calculating additionalstatistics from the number of predicted molecules and locations of thepredicted molecules.

In some examples, the additional statistics include at least one of:percentage of cells having a particular molecule, percentage of cellshaving a particular ratio of molecules, location relationships amongcell types, extent of mixing of cell types, and degree of tumorinfiltration by lymphocytes. In one example, a cell type may beassociated with a particular ratio of molecules, or a range of ratios.

In some examples, method further comprises predicting an immunotherapyresponse of the patient based, at least partially, on the additionalstatistics.

In some examples, the assigning the immunotherapy response classincludes comparing the number of predicted molecules to a threshold foreach molecule.

In some examples, the assigning the immunotherapy response classincludes comparing locations of predicted molecules to molecule locationcriteria.

In some examples, the immunotherapy response class is one of low,medium, and high lymphocyte infiltration.

In some examples, the H&E image is associated with a patient.

In some examples, method further comprises predicting an immunotherapyresponse of the patient, based on the number of predicted molecules andlocations of the predicted molecules and matching with immunotherapytreatment.

In accordance with another embodiment, a method for using a machinelearning model to analyze at least one H&E slide image associated with apatient, comprising: a. scoring H&E slide image for similarity to slideimages associated with immunotherapy responders versus slide imagesassociated with immunotherapy non-responders; and b. comparing the scoreto a threshold.

In some examples, the H&E image is associated with a tumor organoid.

In some examples, method further comprises predicting an immunotherapyresponse of the tumor organoid, based on the number of predictedmolecules and locations of the predicted molecules and predicting drugsensitivity response.

In accordance with another embodiment, a method for using a machinelearning model to analyze at least one hematoxylin and eosin (H&E) slideimage associated with a tumor organoid, the method comprising: a.scoring a H&E slide image for similarity to slide images associated withimmunotherapy responders versus slide images associated withimmunotherapy non-responders; and b. comparing the score of the H&Eslide image to a threshold.

In accordance with another embodiment, a method for generating trainingdata for a histology image-based machine learning model, the methodcomprising: a. obtaining at least one H&E slide image associated with abiological specimen; b. obtaining one or more multipleximmunohistochemistry (IHC) images associated with the biologicalspecimen, wherein each multiplex IHC image includes at least two IHCstains, where each IHC stain has a unique color and a unique targetmolecule; c. for each multiplex IHC image, detecting mixture colorscomprised of more than one IHC stain and identifying the IHC stains thatcomprise each mixture color; d. determining the location of each IHCstain color and determining the location of the associated stainedtarget molecules; e. detecting individual cell locations and determiningwhich individual cells are lymphocytes; f. for each H&E image and IHCimage associated with the biological specimen, align/register imagessuch that for each physical location in the biological specimen, allpixels associated with that physical location are aligned; g. for eachtarget molecule, marking the location on the H&E image that correspondsto the locations of the target molecules stained on the IHC layers; h.for each cell having a location that corresponds to the location of oneor more IHC stains, calculating the percentage of stained pixelsoverlapping the cell that is associated with each IHC stain to determinean IHC stain profile for each cell; and i. storing marked and unmarkedversions of the H&E image as part of a training data set.

In some examples, the H&E image is captured from a tissue layer that isstained only with H&E.

In some examples, the H&E image is captured from a tissue layer that isstained with H&E and at least one IHC stain.

In some examples, the H&E image is a virtual H&E stain image generatedbased on cell and tissue structures visible in a brightfield image of atissue layer.

In some examples, determining the location of each IHC stain colorincludes setting an intensity threshold for each stain color andcomparing the intensity of the stain color in each pixel to theintensity threshold for that stain color.

In some examples, method further comprises generating an overlay foreach IHC stain where each pixel having an intensity that exceeds thethreshold for the IHC stain is annotated to indicate presence of the IHCstain in the pixel.

In some examples, detecting cell locations is performed by a neuralnetwork.

In some examples, detecting cell locations includes the use of UNET.

In some examples, identifying the IHC stains that comprise each mixturecolor is accomplished by deconvolving mixture colors within each image.

In some examples, method further comprises assigning a tissue class toportions of the H&E image.

In some examples, method further comprises associating an immunotherapyresponse score with the stored unmarked H&E image, based on clinicaldata associated with the biological specimen.

In some examples, the immunotherapy response score is based onimmunotherapy associated sequencing data, Immune Cell Infiltration,Immune Gene Expression Signatures, Multiplex PD-L1 and CD8 staining, andMultiplex macrophage IHC panels.

In some examples, the immunotherapy associated sequencing data includestumor mutational burden (TMB), microsatellite Instability (MSI), and TCell Clonality.

In some examples, processes (a)-(i) of the preceding method areperformed for a plurality of biological specimens to generate thetraining data set.

In some examples, method further comprises: e. receiving the biologicalspecimen; f. dividing the biological specimen into a plurality of tissuelayers; g. simultaneously adding at least two classes ofantibody-conjugated (IHC) stain to one of the tissue layers, whereineach class of antibody-conjugated stain binds to a unique class oftarget molecule and each class of antibody-conjugated stain has a uniquestain color, such that each stain color is associated with a targetmolecule; and h. for each of the stained layers, capturing and storingone digital image.

In some examples, the target molecule in a first tissue layer is CD3;the target molecule in a second tissue layer is CD8; the target moleculein a third tissue layer is CD20; the target molecule in a fourth tissuelayer is CD68; a fifth tissue layer is stained with H&E; the targetmolecules in a sixth tissue layer are CD3, CD8, CD20, CD68, CK, PD1, andPDL1; the target molecules in a seventh tissue layer are CD3, CD8, CD20,and CD68; the target molecules in an eighth tissue layer are CK, PD1,and PDL1; the target molecule in a ninth tissue layer is CK; the targetmolecule in a tenth tissue layer is PD1; and the target molecule in aneleventh tissue layer is PDL1.

In some examples, method further comprises simultaneously adding to onetissue layer of the tissue layers, a plurality of IHC stains such thatthe target molecules in the one tissue layer are CD3, CD8, CD20, CD68,CK, PD1, and PDL1.

In accordance with another embodiment, a method for training a histologyimage-based machine learning model, the method comprising: a. receivinga training data set comprising unmarked H&E images and data associatedwith each unmarked H&E image; and b. optimizing the histologyimage-based machine learning model to receive an unmarked H&E image andgenerate a simulated data set similar to the data associated with thatunmarked H&E image.

In some examples, the associated data includes a corresponding markedH&E image for each unmarked H&E image, wherein the marked H&E imageshows the location of IHC staining target molecules in one or more IHCimages associated with the same biological specimen as the H&E image,where at least one of the IHC images is a multiplex IHC image having twoor more IHC stains.

In some examples, the associated data includes an immunotherapy responsescore.

In some examples, method further comprises receiving the training dataset of claim 1.

In some examples, method further comprises receiving the training dataset of claim 14.

In some examples, the histology image-based machine learning model is aneural network.

BRIEF DESCRIPTION OF THE DRAWINGS

This patent or application file contains at least one drawing executedin color. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the United States Patent andTrademark Office upon request and payment of the necessary fee.

The figures described below depict various aspects of the system andmethods disclosed herein. It should be understood that each figuredepicts an example of aspects of the present systems and methods.

FIG. 1 illustrates a prediction system capable of analyzing digitalimages of histopathology slides of a tissue sample, such as H&E slideimages, and determining the likelihood of biomarker presence in thattissue, in accordance with an example.

FIG. 2 illustrates a process of analyzing digital images ofhistopathology slides of a tissue sample using a trained machinelearning model as may be performed by the prediction system of FIG. 1,in accordance with an example.

FIG. 3 illustrates an example report in the form of a cancer immunogramscore that may be included in a patient report as generated by theprocess of FIG. 2, in accordance with an example.

FIG. 4 illustrates a system for generating training data and fortraining a histology image-based machine learning model, in accordancewith an example.

FIG. 5 illustrates an example implementation of a system for generatingtraining data and for training a histology image-based machine learningmodel, in accordance with an example.

FIG. 6A-6C illustrates an example neural network architecture used fortissue classification and having a fully convolution networkarchitecture; FIG. 6B illustrates example output sizes for differentlayers and resulting sublayers of the architecture of FIG. 6A; and FIG.6C is a visualization of each depth of an exemplary 3-dimensional inputimage matrix being convoluted by two exemplary 3-dimensional filtermatrices using the architecture of FIG. 6A, for example, as may beimplemented by the system of FIG. 4, in accordance with an example.

FIG. 7 illustrates an example neural network architecture used fortissue classification and having a UNet architecture, in accordance withan example.

FIG. 8 illustrates example H&E images at different stages of processingby a prediction system, in accordance with an example.

FIG. 9 illustrates an example H&E image resulting from a predictionsystem and showing lymphocyte cells detected within tumor tiles (TILs)determined by the prediction system, in accordance with an example.

FIG. 10 illustrates an immunotherapy response decision system capable ofdetermining a listing of possible immunotherapies based on receiveddata, in accordance with an example.

FIG. 11 illustrates an example system having a dedicated ultrafastpathology (slide) scanner system and an imaging-based biomarkerprediction system, in accordance with an example.

FIG. 12 illustrates an example computing device for implementing theimaging-based biomarker prediction system of FIGS. 1 and 11 and othersystems herein, in accordance with an example.

DETAILED DESCRIPTION

In various embodiments, the systems and methods disclosed hereinannotate digital images of hematoxylin and eosin (H&E) slides withlocations of immunohistochemistry (IHC) targets located in IHC stainedhistology slides associated with the same biological specimen as the H&Eslide. In various embodiments, the systems and methods include automatedprocesses for adding IHC labels from multiplexed IHC slide images to H&Eslide images.

Multiplex imaging, such as used with multiplexed IHC slide images,allows for the assessment of multiple biomarkers from a single sample,or single slide image. In many examples, the ability to identifymultiple biomarkers from a single slide is greatly advantageous overattempting to use multiple adjacent slides to detect multiplebiomarkers. Different slides, even adjacent, often do not share the samecells. Moreover, even tissue may not be shared fully across adjacentslides. Indeed, by providing techniques capable of identifying andlabeling locations of multiple IHC stains to be applied to H&E slideimages, numerous advantages result. The present techniques can visualizemultiple target antigens within a single tissue sample and even a singleslice of a tissue sample, thereby maximizing the amount of data acquiredfrom a single tissue sample. The present techniques can examine thespatial arrangements, interactions, and co-localizations of multipleproteins in a single H&E slide image. Moreover, unlike next generationsequencing techniques, image based analyses herein do not destroy tissuesamples to test for individual target molecules.

In various embodiments, systems and methods are provided for using amachine learning model to analyze H&E slide images to predict locationsof cells and/or molecules in the H&E slide image. Such predictions canbe made alongside identification of tissue and other structures in theH&E image. In some embodiments, the machine learning models are trainedusing a training data set that includes (i) unmarked H&E images and (ii)marked H&E images, where those marked H&E images include location dataidentifying one or more molecules and where that location data isdetermined from an analysis of multiplex IHC images. The multiplex IHCimages have at least two IHC stains, each IHC stain having a unique(i.e., different) color and a unique (i.e., different) target molecule.Moreover, the analysis of those multiplex IHC images includesdetermining which IHC stain(s) contribute to two or moreoverlapping/adjacent IHC stains thereby allowing for moleculeidentification across different stains. With the machine learning modelstrained using multiplex IHC, in this way, these models may then be usedto analyze molecules and locations in H&E slide images and assign animmunotherapy response class to the H&E slide image.

In various embodiments, the H&E slide images fed to trained machinelearning models can be virtual, e.g., based on a brightfield image. Insome examples, the H&E slide images are generated by adding H&E stainsto a tissue slice having multiple IHC stains, on top of the mIHC stains.In some examples, a tissue slice with multiple IHC stains is washed toremove the IHC stains and then H&E stains are added to the tissue sliceto generate the H&E slide images.

In various embodiments, the machine learning models herein are trainedusing ground truth annotated images, in particular ground truthsdeveloped from multiplex IHC images. The recent advance of computationalpathology systems has enabled the identification of immune-relatedbiomarkers (termed immunotherapy biomarkers) and a detailedunderstanding of the tumor microenvironment. Examples of biomarkersbeing explored in immunotherapy (immunotherapy biomarkers) include serumproteins, tumor-specific receptor expression patterns, factors in thetumor microenvironment, circulating immune and tumor cells, and hostgenomic factors. Such systems can identify and quantify different immunecell subsets, their spatial context, and the expression of immunecheckpoint markers, albeit with degrees of accuracy. Buildingcomputational pathology systems is facilitated by having large-scale,high-quality ground truth annotations.

Therefore, in various embodiments, the systems and methods hereininclude generating ground truth annotated images. In some examples,these techniques include staining, digitizing, and analyzing brightfieldmultiplex (Mx) FFPE tissue slides to generate ground truth annotations.More generally, systems and methods may generate multiplex-derivedannotations, which may be used to train and quantitatively validate amachine learning model trained for histology tissue segmentation andcell detection of H&E slide images.

FIG. 1 illustrates a prediction system 100 capable of analyzing digitalimages of histopathology slides of a tissue sample, such as H&E slideimages, and determining the likelihood of biomarker presence in thattissue, where biomarker presence indicates a predicted tumor presence, apredicted tumor state/condition, or other information about a tumor ofthe tissue sample, such as a possibility of clinical response throughthe use of a treatment associated with the biomarker or predicted tumorstate/condition.

The system 100 includes an imaging-based biomarker prediction system 102that implements, among other things, image processing operations, deeplearning frameworks, and report generating operations to analyzehistopathology images of tissue samples and predict the presence ofbiomarkers in the tissue samples. In various examples, the system 100 isconfigured to predict the present of these biomarkers, tissuelocation(s) associated with these biomarkers, and/or cell location ofthese biomarkers.

The imaging-based biomarker prediction system 102 may be implemented onone or more computing devices, such as a computer, tablet or othermobile computing device, or server, such as a cloud server. Theimaging-based biomarker prediction system 102 may include a number ofprocessors, controllers or other electronic components for processing orfacilitating image capture, generation, or storage and image analysis,and deep learning tools for analysis of images, as described herein. Anexample computing device 1000 for implementing the imaging-basedbiomarker prediction system 102 is illustrated in FIG. 12.

As illustrated in FIG. 1, the imaging-based biomarker prediction system102 is connected to one or more medical data sources through a network104. The network 104 may be a public network such as the Internet,private network such as a research institution's or corporation'sprivate network, or any combination thereof. Networks can include, localarea network (LAN), wide area network (WAN), cellular, satellite, orother network infrastructure, whether wireless or wired. The network 104can be part of a cloud-based platform. The network 104 can utilizecommunications protocols, including packet-based and/or datagram-basedprotocols such as internet protocol (IP), transmission control protocol(TCP), user datagram protocol (UDP), or other types of protocols.Moreover, the network 104 can include a number of devices thatfacilitate network communications and/or form a hardware basis for thenetworks, such as switches, routers, gateways, access points (such as awireless access point as shown), firewalls, base stations, repeaters,backbone devices, etc.

Via the network 104, the imaging-based biomarker prediction system 102is communicatively coupled to receive medical images, for example ofhistopathology slides such as digital H&E stained slide images,multiplex and single-plex IHC stained slide images, or digital images ofany other staining protocols from a variety of different sources. Thesesources may include a physician clinical records systems 106 and ahistopathology imaging system 108. Any number of medical image datasources could be accessible using the system 100. The histopathologyimages may be images captured by any dedicated digital medical imagescanners, e.g., any suitable optical histopathology slide scannerincluding 10×, 20×, and/or 40× resolution magnification scanners.Further still, the biomarker prediction system 102 may receive imagesfrom histopathology image repositories 110. In yet other examples,images may be received from a partner genomic sequencing system 112,e.g., the TCGA and NCI Genomic Data Commons. In the illustrated example,the biomarker prediction system 102 may receive multiplex IHC imagesfrom an imaging source 116. These image sources may communicate imagedata, genomic data, patient data, treatment data, historical data, etc.,in accordance with the techniques and processes described herein. Eachof the image sources may represent multiple image sources. Further, eachof these image sources may be considered a different data source, thosedata sources may be capable of generating and providing imaging datathat differs from other providers, hospitals, etc. The imaging databetween different sources potentially differs in one or more ways,resulting in different data source-specific bias, such as in differentdyes, biospecimen fixations, embeddings, staining protocols, anddistinct pathology imaging instruments and settings.

In the example of FIG. 1, the imaging-based biomarker prediction system102 includes an image pre-processing sub-system 114 that performsinitial image processing to enhance image data for faster processing intraining a machine learning framework and for performing biomarkerprediction using a trained deep learning framework. Such imagepre-processing may include performing a normalization process onreceived image data, including one or more of color normalization 114 a,intensity normalization 114 b, and imaging source normalization 114 c,to compensate for and correct for differences in the received imagedata. While in some examples the imaging-based biomarker predictionsystem 102 receives medical images, in other examples the sub-system 114is able to generate medical images, including multiplex IHC images,either from received histopathology slides or from other receivedimages, such as generating composite histopathology images by aligningshifted histopathology images to compensate for vertical/horizontalshift. This image pre-processing allows a deep learning framework tomore efficiently analyze images across large data sets (e.g., over1000s, 10000s, to 100000s, to 1000000s of medical images), therebyresulting in faster training and faster analysis processing.

The image pre-processing sub-system 114 may perform further imageprocessing that removes artifacts and other noise from received imagesby doing tile mask application and tile-specific tissue detection 114 d,for example, to identify regions of the images corresponding to tissuefor subsequent analysis, classification, and segmentation. For example,as further described herein, in multiscale configurations where imagedata is to be analyzed on a tile-basis, the masking & detectionsub-system 114 d may receive an initial histopathology image, at a firstimage resolution, downsampling that image to a second image resolution,and then performing a normalization on the downsampled histopathologyimage, such as color and/or intensity normalization, and removingnon-tissue objects from the image, and then apply a tile mask togenerate tiles representing sub-sections of the received images.

The imaging-based biomarker prediction system 102 may be a standalonesystem interfacing with the external (i.e., third party)network-accessible systems 106, 108, 110, 112, and 116. In someexamples, the imaging-based biomarker prediction system 102 may beintegrated with one or more of these systems, including as part of adistributed cloud-based platform. For example, the system 102 may beintegrated with a histopathology imaging system, such as a digital H&Estain imaging system, e.g. to allow for expedited biomarker analysis andreporting at the imaging station. Indeed, any of the functions describedin the techniques herein may be distributed across one or more networkaccessible devices, including cloud-based devices.

In some examples, the imaging-based biomarker prediction system 102 ispart of a comprehensive biomarker prediction, patient diagnosis, andpatient treatment system. For example, the imaging-based biomarkerprediction system 102 may be coupled to communicate predicted biomarkerinformation, tumor prediction, and tumor state information to externalsystems, including a computer-based pathology lab/oncology system 118that may receive a generated biomarker report including image overlaymapping and use the same for further diagnosing cancer state of thepatient and for identifying matching therapies for use in treating thepatient. The imaging-based biomarker prediction system 102 may furthersend generated reports to a computer system 120 of the patient's primarycare provider and to a physician clinical records system 122 fordatabasing the patients report with previously generated reports on thepatient and/or with databases of generated reports on other patients foruse in future patient analyses, including deep learning analyses, suchas those described herein.

To analyze the received histopathology image data and other data, theimaging-based biomarker prediction system 102 includes a deep learningframework 150 that implements various machine learning techniques togenerate trained classifier models for image-based biomarker analysisfrom received training sets of image data or sets of image data andother patient information. With trained classifier models, the deeplearning framework 150 is further used to analyze and diagnose thepresence of image-based biomarkers in subsequent images collected frompatients. In this manner, images and other data of previously treatedand analyzed patients is utilized, through the trained models, toprovide analysis and diagnosis capabilities for future patients.

In various embodiments, the deep learning framework 150 includes amultiplex histopathology image-based classifier training module 160 thatcan access received and stored images and data from the external systems106, 108, 110, 112, and 116, and any others, where that data may beparsed from received data streams and databased into different datatypes. The different data types may be divided into image data 162 awhich may be associated with the other data types molecular data 162 b,demographic data 162 c, and tumor response data 162 d. An associationmay be formed by labeling the image data 162 a with one or more of thedifferent data types. By labeling the image data 162 a according toassociations with the other data types, the imaging-based biomarkerprediction system may train an image classifier module to predict theone or more different data types from image data 162 a.

In various embodiments herein, and as discussed further herein, theimage data 162 a may be multiplex histology images, such as one or moremultiplex IHC stain images. As discussed further herein, the multiplexclassifier training module 160 may include a multiplex image deconvolver161, a stain image target identifier 163, and a cell identifier neuralnetwork 165.

In the illustrated data, the deep learning framework 150 includes imagedata 162 a. For example, to train or use a multiscale PD-L1 biomarkerclassifier, this image data 162 a may include pre-processed image datareceived from the sub-system 114, images from H&E slides or images fromIHC slides (with or without human annotation), including IHC slidestargeting PD-L1. The IHC slides may include additional targets, forexample, PTEN, EGFR, Beta catenin/catenin betal, NTRK, HRD, PIK3CA, andhormone receptors including HER2, AR, ER, and PR. To train or use otherbiomarker classifiers, whether multiscale classifiers or single-scaleclassifiers, the image data 162A may include images from other stainedslides. Further, in the example of training a single scale classifier,the image data 162A is image data associated with RNA sequence data forparticular biomarker clusters, to allow the multiple instance learning(MIL) techniques herein.

The molecular data 162 b may include DNA sequences, RNA sequences,metabolomics data, proteomic/cytokine data, epigenomic data, tumororganoid data, raw karyotype data, transcription data, transcriptomics,microbiomics, and immunomics, identification of variants (for example,SNP, MNP, InDel, microsatellite Instability (MSI), tumor mutationalburden (TMB), CNV Fusions, loss of heterozygosity, loss or gain offunction). Epigenomic data includes DNA methylation, histonemodification, or other factors which deactivate a gene or causealterations to gene function without altering the sequence ofnucleotides in the gene. Microbiomics includes data on viral infectionswhich may affect treatment and diagnosis of certain illnesses as well asthe bacteria present in the patient's gastrointestinal tract which mayaffect the efficacy of medicines ingested by the patient or otherclinical characteristics of the patient. Proteomic data includes proteincomposition, structure, and activity; when and where proteins areexpressed; rates of protein production, degradation, and steady-stateabundance; how proteins are modified, for example, post-translationalmodifications such as phosphorylation; the movement of proteins betweensubcellular compartments; the involvement of proteins in metabolicpathways; how proteins interact with one another; or modifications tothe protein after translation from the RNA such as phosphorylation,ubiquitination, methylation, acetylation, glycosylation, oxidation, ornitrosylation.

The deep learning framework 150 may further include demographic data 162c and tumor response data 162 d (including data about a reduction in thegrowth of the tumor after exposure to certain therapies, for exampleimmunotherapies, DNA damaging therapies like PARP inhibitors orplatinums, or HDAC inhibitors). The demographic data 162 c may includeage, gender, race, national origin, etc. The tumor response data 162 dmay include epigenomic data, examples of which include alterations inchromatin morphology and histone modifications.

The tumor response data 162 d may include cellular pathways, example ofwhich include IFNgamma, EGFR, MAP KINASE, mTOR, CYP, CIMP, and AKTpathways, as well as pathways downstream of HER2 and other hormonereceptors. The tumor response data 162 d may include cell stateindicators, examples of which include Collagen composition, appearance,or refractivity (for example, extracellular vs fibroblast, nodularfasciitis), density of stroma or other stromal characteristics (forexample, thickness of stroma, wet vs. dry) and/or angiogenesis orgeneral appearance of vasculature (including distribution of vasculaturein collagen/stroma, also described as epithelial-mesenchymal transitionor EMT). The tumor response data 162 d may include tumorcharacteristics, examples of which include the presence of tumor buddingor other morphological features/characteristics demonstrating tumorcomplexity, tumor size (including the bulky or light status of a tumor),aggressiveness of tumor (for example, known as high grade basaloidtumor, especially in colorectal cancer, or high grade dysplasia,especially in barrett's esophagus), and/or the immune state of a tumor(for example, inflamed/“hot” vs. non-inflamed/“cold” vs immuneexcluded).

The multiplex histopathology image-based classifier training module 160may be configured with an image-analysis adapted machine learningtechniques, including, for example, deep learning techniques, including,by way of example, a CNN model and, more particular, a tile-resolutionCNN, that in some examples is implemented as a FCN model, and, moreparticularly still, implemented as a tile-resolution FCN model. Any ofthe data types 162 a-162 d may be obtained directly from datacommunicated to the imaging-based biomarker prediction system 102, suchas contained within and communicated along with the histopathologyimages. The data types 162 a-162 d may be used by the histopathologyimage-based classifier training module 160 to develop classifiers foridentifying one of more of the biomarkers discussed herein, such asimmunotherapy biomarkers.

In the example system 100, the deep learning framework 150 furtherincludes a trained image classifier module 170 that may also beconfigured with the deep learning techniques, including thoseimplementing the module 160. In some examples, the trained imageclassifier module 170 accesses the image data 162 for analysis andbiomarker classification. In some examples, the module 170 furtheraccesses the molecular data 162, the demographic data 162 c, and/ortumor response data 162 d for analysis and tumor prediction, matchedtherapy predictions, etc.

The trained image classifier module 170 includes trained tissueclassifiers 172, trained by using ground truth mask images and otherdata from the module 160, using one or more training image sets, toidentify and classify tissue type in regions/areas of received imagedata.

The module 170 also includes trained cell classifiers 174 that identifyimmunotherapy biomarkers via cell classification in received histologyimages, e.g., in unmarked H&E slide images. The module 170 may furtherinclude a cell segmenter 176 that identifies cells within ahistopathology image, including cell borders, interiors, and exteriors,albeit in some examples, the cell segmentation is a trained model of thecell classifier 174.

In examples herein, the tissue classifiers 172 may include biomarkerclassifiers specifically trained to identify tumor infiltration (such asby ratio of lymphocytes in tumor tissue to all cells in tumor tissue),lymphocyte infiltration (high, medium, or low), PD-L1 (such as positiveor negative status), ploidy (such as by a score), CMS (such as toidentify subtype), NC Ratio (such as nucleus size identification),signet ring morphology (such as a classification of a signet cell orvacuole size), HRD (such as by a score, or by a positive or negativeclassification), etc. in accordance with the immunotherapy biomarkersherein. It will be appreciated that lymphocyte infiltration is a type oftumor infiltration.

As detailed herein, the trained image classifier module 170 andassociated classifiers may be configured with an image-analysis adaptedmachine learning techniques, including, for example, deep learningtechniques, including, by way of example, a CNN model and, moreparticular, a tile-resolution CNN, that in some examples is implementedas a FCN model, and, more particularly still, implemented as atile-resolution FCN model, etc.

The system 102 further includes a tumor report generator 180 configuredto receive classification data from the trained tissue (biomarker)classifiers 172, the trained cell (biomarker) classifiers 174 and thecell segmenter 172 and determine tumor metrics for the image data andgenerate digital image and statistical data reports, where such outputdata may be provided to the pathology lab 118, primary care physiciansystem 120, genomic sequencing system 112, a tumor board, a tumor boardelectronic software system, or other external computer system fordisplay or consumption in further processes. In various examples, thetumor report generator 180 receives the classification data andcalculates various statistics, including one or more of percentage ofcells having a particular molecule, percentage of cells having aparticular ratio of molecules, location relationships among cell types,extent of mixing of cell types, degree of tumor infiltration bylymphocytes (high, medium, or low), and/or other statistics herein. Thereport generator 180 may calculate these statistics from the number ofpredicted molecules and locations of the predicted molecules containedin the classification data. In some implementations, these statisticsare determined by a machine learning model, such as machine learningmodel 304.

In various embodiments, the imaging-based biomarker prediction system102 is configured to train machine learning models to analyze H&E slideimages, where that training is based on received multiplex IHC stainedimages. In an example, a process 200, in FIG. 2, is provided fortraining a machine learning model and analyzing H&E slide images usingthat trained machine learning to assist in determination of animmunotherapy response for a subject. In the illustrated example, theprocess 200 is implemented in two stages, an optional multiplex IHCstaining stage 202 and a machine learning training stage 204.

In an example implementation of the multiplex IHC staining stage 202, ata process 206 a biological specimen is received and histology slides aregenerated. The biological specimen may be taken from a biopsy, bonemarrow biopsy, endoscopic biopsy, needle biopsy (e.g., fine-needleaspiration, core needle, vacuum-assisted, image-guided), skin biopsy(e.g., shave, punch, incisional, excisional), surgical biopsy, etc. Theprocess 206 may further include dividing the received biologicalspecimen into a plurality of tissue layers, known as slices, where theslices may be approximately 4-13 microns thick.

At a process 208, a staining protocol is selected for use on each of theslices. In various embodiments, the staining protocol includesidentification of the biological specimens that will be stained, such asspecimen tissue source, organ source, etc. and/or other data. Thestaining protocol may further include identification of the type ofstaining equipment (for example, stainer) to be used including strainerscapable of multiplex IHC staining. The staining protocol may furtherinclude IHC targets for each tissue slice (or if the tissue slice willbe H&E stained), the antibody to use for each IHC target, the chromogento use for each IHC target, and the order in which each target will bestained (for multiplexed IHC staining). Conventional slice preparationtechniques may be included, as well as post-staining steps for imaginganalysis. In various embodiments, IHC targets may be various molecules,for example, proteins, RNA, DNA, lipids, sugars, or any molecule thatcan be bound by a stain-conjugated antibody or stained by other means ofcellular staining. In some examples, the IHC staining protocol isselected based on any of the foregoing data. In various embodiments, atthe process 208, the IHC staining protocol is selected and optimized. Anexample optimization is described further according to Example 1.

In an embodiment, one of the slices from process 206 is stained with H&Eand at least one of the other slices is stained with multiple classes ofIHC antibodies, to form an multiplex IHC (mIHC) slide. To generate themultiplex slide, at least two classes of antibody-conjugated (IHC) stainare applied to a tissue layer, wherein each class of antibody-conjugatedstain binds to a unique class of target molecule and each class ofantibody-conjugated stain has a unique stain color, such that each staincolor is associated with a target molecule. In various examples,preferably 2-10 antibody-conjugated IHC stains are applied to form amIHC stain. That is, in various examples 2, 3, 4, 5, 6, 7, 8, 9, or 10IHC stains may be applied to a slice to form an mIHC stain slice.Optionally, to include additional slice images in the training dataset,for each remaining slice, at least one class of antibody-conjugatedstain is added to the slice. In some examples, the training dataset maybe based on having multiple multiplex IHC stain images.

In this way, in various embodiments, the systems and methods addmultiple antibodies to the same slide simultaneously, at a process 210at which the slices are stained according to the selected IHC stainingprotocol. If the antibodies are from different species, there may beintermediate steps when adding these antibodies at process 210. Invarious embodiments, however, the addition of DAB or heptane to blockantibodies between any intermediate steps may not be required. Moreover,by using multiple IHC staining, techniques herein do not require cyclicstaining and are likely to cause tissue degradation. That is, unlikeconventional techniques that require various cyclic immunofluorescence(IF) and immunohistochemical (IHC) methods using cycles of fluorescenttagging, imaging and bleaching or dissociation of the affinity tags toimprove spectral resolution of immunostaining, the present techniquescan train machine learning models without needing to serially image asample that is re-stained many times.

While IHC staining protocol selection may be partially manuallyperformed, in some examples, the process 210 applies an automatedprocedure to stain the slices according to the staining protocol.Automation allows standardizing the staining process to reducevariability and improve staining quality, reproducibility, and speed.For example, this may include using a Ventana DISCOVERY ULTRA Researchautostainer. (see

https://diagnostics.roche.com/us/en/products/instruments/discovery-ultra.html).

In the illustrated example, at a process 212, the staining slides arescanned using a slide imager and the resulting H&E slide image andmultiplex IHC slide image(s) and any single IHC slide images areanalyzed using image processing to perform an image quality assessment.In an example, slide images may be scanned at 40× magnification with thePhilips IntelliSite Pathology Solution Ultra Fast Scanner. Slide imagesthat do not pass image quality assessment post-scan (for example, slideimages that are out of focus) will be rejected by the process 212,discarded, and the imager automatically instructed to rescan the slide.Slide images that satisfy the image quality assessment, will be passedto the machine learning training process 204.

Therefore, in the illustrated example, the processes 206-212 form theIHC staining process 202 from which control is passed to the machinelearning training process 204. Alternatively, multiplex IHC image(s),H&E slide image(s), and single IHC image(s) of scanned slides may bereceived directly to the process 204 and the optional procedures ofprocess 202 avoided.

To train machine learning models, the process 204 automatically locatesIHC targets within mIHC slide images to generate training data. At aprocess 214, histology slide images associated with the same biologicalspecimen are aligned, including the H&E and multiplex IHC stain images.In various embodiments, the process 214 register the histology images,by digitally aligning images such that they reflect the relative3-dimensional position of the tissue before the specimen was sliced intotissue layers. In an example, the process 214 may overlay a digitallocation grid on each histology image (i.e., on each H&E stain andmultiple IHC and single IHC associated with the sample) such that alocation coordinate on that grid is associated with corresponding(stacked) locations on all slides for a biological specimen. Next, theprocess 214 may apply a global alignment. In an example, all IHC slides(single and multiplex) may be co-registered with the H&E tissue slide. Atissue segmentation may be performed and then a global rigidregistration (e.g., an affine transform) may be applied to consecutiveslides to align tissues at a sufficient threshold magnification, such asat 10× magnification. Further still, at the process 214 a localalignment (e.g., a per-patch non-linear registration) may be applied ata sufficient threshold magnification, typically larger than the rigidregistration magnification (e.g., 20× magnification), using normalizedgradient fields. This local alignment may be useful for cell-to-cellcomparisons across serial sections in the histology images, to perform acell-level registration. In some embodiments, the registrationalgorithm's performance may be determined via a qualitative review ofdifferent field-of-views randomly sampled within the H&E tissue region.

In various embodiments, location and molecular and cell data isdeveloped across the different histology images to develop the trainingdataset. In the illustrated example, a process 216 determines thelocation of ICH stained targets in all histology slide images, includingin the H&E slide images, and by using a deconvolving process to resolvecells and molecules co-extensive across multiple IHC stains. Whethermultiplex or single stain, IHC stained targets include molecules, cellmarkers, other cell components. The process 216 identifies the multiplexIHC images and deconvolves any overlapping stain colors. That is,multiple colors may overlap on IHC stained slides having more than oneIHC target/multiplexed IHC targets. Color deconvolution facilitatesdetection of two or more co-localized antibodies (for example, two ormore antibodies located in close proximity or in the same location on aslide) and thus the detection of two or more co-localized targetmolecules bound to the antibodies. An example, deconvolution process isdescribed in Haub, P., Meckel, T. A, Model based Survey of ColourDeconvolution in Diagnostic Brightfield Microscopy: Error Estimation andSpectral Consideration, Sci. Rep 5, 12096 (2015)

(https://doi.org/10.1038/srep12096), the contents of which areincorporated herein by reference in their entirety.

In various embodiments, to perform deconvolution, the process 216 isconfigured to determine spectral absorbance/transmittance values atdifferent wavelengths and across the multiplex IHC image. In particular,the process 216 is configured to apply a procedure built upon theBouguer-Lambert-Beer equation, which describes the absorption ofmonochromatic radiation passing absorbing dyes:

l(λ)=l ₀(λ)·e ^(−δ(λ)·) c  (1)

where l₀(λ) is the spectral radiation intensity, l(λ) is the transmittedspectral intensity, δ(λ) is the spectral molar optical density for aunified layer thickness, and c is the dye concentration.

For two or more stains i located close enough to each other that theycombine to appear as a mixture color, a spectral absorbance A(λ) can beexpressed as:

$\begin{matrix}{{A(\lambda)} = {{- {\ln\left( \frac{I(\lambda)}{I_{0}(\lambda)} \right)}} = {\sum_{i}\left( {{\delta_{i}(\lambda)} \cdot c_{i}} \right)}}} & (2)\end{matrix}$

With known spectral absorbance values Ajip for different pure stains i,a set of linear equations can be formulated to express the spectralabsorbance Aj for different wavelength j, e.g., for two wavelengths kand l and two pure stains, as:

A _(k) =A _(k1p) ·c′ ₁ +A _(k2p) ·c′ ₂  (3)

A _(l) =A _(l1p) ·c′ ₁ +A _(l2p) ·c′ ₂  (4)

Thus, in some examples, the process 216 uses these linear equations to‘unmix’ measured absorbance values Aj by calculating the relativeconcentrations c′1 and c′2. This approach can be extended to higherorders of wavelengths and stains, i.e., where more than 2 stains areused to form the multiplex IHC. In some examples, the process 216 mayreceive IHC staining data, such as from the IHC staining protocolidentified at 208. In some examples, the process 216 may apply abrute-force method of analyzing receiving IHC images at multipledifferent wavelengths, assessing spectral absorption/transmittance fromwhich the process identifies between multiplex IHC images and single IHCimages and from which the process determines the number of IHC stains inthe multiplex IHC image and the stain types (including associatedabsorption wavelengths).

A stain vector {right arrow over (A_(lp))} describes the absorbancecharacteristics of a pure stain i and is expressed, for example, for twomonochromatic wavelengths k and l by the stains spectral transmittanceτp(λ):

$\begin{matrix}{\overset{\rightarrow}{A_{ip}} = {\begin{pmatrix}A_{kip} \\A_{lip}\end{pmatrix} = {\begin{pmatrix}{- {\ln\left( \frac{I_{ip}\left( \lambda_{k} \right)}{I_{0}\left( \lambda_{k} \right)} \right)}} \\{- {\ln\left( \frac{I_{ip}\left( \lambda_{l} \right)}{I_{0}\left( \lambda_{l} \right)} \right)}}\end{pmatrix} = \begin{pmatrix}{- {\ln\left( {\tau_{ip}\left( \lambda_{k} \right)} \right)}} \\{- {\ln\left( {\tau_{ip}\left( \lambda_{l} \right)} \right)}}\end{pmatrix}}}} & (5)\end{matrix}$

Stain vectors define the target coordinate system for the lineartransformation from absorbance into concentration space. The process216, thus, may specify stain vectors prior to the deconvolution, wherethese stain vectors can be estimated from samples ideally stained withpure dyes.

If normalized stain vectors with a unit length of 1 are used forunmixing, the resulting normalized relative concentration values c* arerelated to relative concentration c′ by:

$\begin{matrix}{c_{i}^{\prime} = \frac{c*i}{\overset{\rightarrow}{A_{ip}}}} & (6)\end{matrix}$

In some diagnostic brightfield applications of the techniques herein,the absorbance values used for stain vector estimation and colordeconvolution (CD), are calculated from the sensor signals VR, VG, VBmeasured with scientific RGB color cameras in a slide imager. For atypical RGB camera with 8 bit maximum color channel values V0R, V0G, V0Bthis can be formulated (without considering any disruptive imagingeffects) as:

$\begin{matrix}{A_{R} = {{- {\ln\left( \frac{V_{R}}{V_{0R}} \right)}} = {- {\ln\left( \frac{V_{R}}{255} \right)}}}} & (7) \\{A_{G} = {{- {\ln\left( \frac{V_{G}}{V_{0\; G}} \right)}} = {- {\ln\left( \frac{V_{G}}{255} \right)}}}} & (8) \\{A_{B} = {{- {\ln\left( \frac{V_{B}}{V_{0\; B}} \right)}} = {- {\ln\left( \frac{V_{B}}{255} \right)}}}} & (9)\end{matrix}$

In some embodiments, the process 216 models the formation ofnon-monochromatic camera signals V′R, V′G, V′B by summation of spectralproducts of light intensity Irel(λ), stain transmittance τp(λ) and thesensor characteristics sR(λ), sG(λ), sB(λ). For two stains, for example,this modeling is as follows:

$\begin{matrix}{V_{R}^{\prime} = {\sum\limits_{j = {1\;\ldots\; 60}}\left( {{I_{rel}\left( \lambda_{j} \right)} \cdot {\tau_{1p}\left( \lambda_{j} \right)}^{c_{1}} \cdot {\tau_{2\; p}\left( \lambda_{j} \right)}^{c_{2}} \cdot {s_{R}\left( \lambda_{j} \right)}} \right)}} & (10) \\{V_{G}^{\prime} = {\sum\limits_{j = {1\;\ldots\; 60}}\left( {{I_{rel}\left( \lambda_{j} \right)} \cdot {\tau_{1p}\left( \lambda_{j} \right)}^{c_{1}} \cdot {\tau_{2p}\left( \lambda_{j} \right)}^{c_{2}} \cdot {s_{G}\left( \lambda_{j} \right)}} \right)}} & (11) \\{V_{B}^{\prime} = {\sum\limits_{j = {1\;\ldots\; 60}}\left( {{I_{rel}\left( \lambda_{j} \right)} \cdot {\tau_{1}\left( \lambda_{j} \right)}^{c_{1}} \cdot {\tau_{2}\left( \lambda_{j} \right)}^{c_{2}} \cdot {s_{B}\left( \lambda_{j} \right)}} \right)}} & (12)\end{matrix}$

with λ1 . . . λ60={405 nm, 410 nm, . . . 700 nm}.

Maximum camera values V′0R, V′0G, V′0B were calculated without staining(c′1=c′2=0). Based on these equations the non-linear signal formation issimulated to evaluate the deconvolution of absorbance values derivedfrom these signals.

After deconvolution, the IHC slide image data is analyzed to identifythe location of IHC stained targets in the multiplex IHC image,including IHC stained targets identified by each different IHC staintype of the IHC stained targets coinciding with multiple stain types.That is, the deconvolution is used to determine an IHC stain thatcontributes to any two or more overlapping/adjacent IHC stains. Next, ata process 218, a map of the locations of each instance of the IHC staintarget is determined. For example, each IHC stain in the multiplex IHCimage may be compared to one or more threshold to determine locations ofIHC stained targets. From this, the predicted IHC stain targets, e.g.,the predicted molecules, are identified along with their location in theIHC stain images.

In various embodiments, the process 218 apply a binary mask at the pixellevel. For example, each pixel may be assigned a value of “1” indicatingthe pixel as the color associated with the IHC target or “0” indicatingthe pixel does not have a staining color. A wavelength specificthreshold absorption value may be applied as the binary mask. In someembodiments, the process 218 determines a field of view (FOV) of thehistology images before apply the binary mask. For example, a FOV may beidentified that contains a minimum number of cells, where that minimumnumber has been selected by the user through a graphical user interface.In some examples, binary masks are obtained for each marker using colordeconvolution and intensity-based thresholding on a select set of fieldsof view (FOVs). Each FOV should contain a minimum of x stained cells,where x may be 5, 10, 15, or 20, for example. Further still, in someexamples, the threshold applied by the binary masks may be adjusted by auser through a graphic user interface.

In various embodiments, in addition to the deconvolution and mappinganalysis of the stain overlapping regions of an multiplex IHC describedabove, binary masks may be determined from analyzing the single stain(non-overlapping) regions of these images or of single IHC stain images.For example, manually corrected masks may be obtained from single stainsections of IHC images and applied to a machine learning model topredict multiclass masks from multiplex IHC images.

With the multiplex IHC images mapped and locations of IHC stain targetsidentified across single stain and overlapping stain regions of themultiplex IHC image, at a process 220, a machine learning model isselected and train to analyze H&E input slides for predicting moleculesand molecule location in those H&E slides, for ultimately assigning animmunotherapy response class to the H&E slide image.

In various embodiments, at the process C20, a machine learning model maybe trained for cell detection, cell classification, and/or tissuesegmentation of digital H&E stained slide images. In an example,co-registered H&E and multiplex IHC images, or FOVs, are used to trainan H&E cell detection model with ground truth masks derived from themultiplex IHC slides images. Further still, the, the machine learningmodel may be trained to detect and ignore biological artifacts (forexample, necrotic cells, mucin, etc.). While shown as part of theprocess 200, in some implementations the training of the machinelearning model may be performed separately from the process 200.

With the machine learning model trained, at a process 222 the trainingmachine learning model is used to analyze one or more H&E slide images.For example, at the process 222, a new H&E slide image associated withpatient biopsy, blood sample, or tumor organoid is received and analyzewith trained machine learning model.

The process 222 may be configured to report results of that analysis,where the report may include a digital visual indication of thelocations of target molecules, immune and/or cancer cell types in slideimage. In some examples, the report may include a degree of mixing ofcell types, tumor infiltration, and/or immune infiltration detected inslide image. In some examples, the report may include data on thelikelihood that the patient or tumor organoid will respond toimmunotherapy, i.e., the assigning of an immunotherapy response class.In examples herein, immunotherapy may include ipilimumab, atezolizumab,avelumab, durvalumab, nivolumab, pembrolizumab, anti-PD-L1 therapies,checkpoint inhibitors, axicabtagene ciloleucel, tisagenleleucel, CART-cell therapies, any combination thereof, etc. For example,immunotherapies are described inhttps://www.archivesofpathology.org/doi/full/10.5858/arpa.2018-0584-CP,the contents of which are incorporated herein by reference in theirentirety for any and all purposes.

In various applications the process 200 can be used for early stageclinical trials, where histology image data for training is relativelysmall. By using multiplex IHC stain images and the deconvolutionprocesses here, allowing for identification of molecules and molecularlocation in overlapping stain regions, a few number of tissue slices maybe used to train a robust machine learning model. Further that multiplexIHC stain images are used to train a machine learning model operating onH&E stain images, allows for faster analysis and immunology responseprediction, through the ability to identify not on conventionalstructural features in an H&E image but various different types ofmolecules and molecule location. Thus, the process 200 allows forpharmacologic research to be deployed with machine learning techniques,even at the earliest stages to identify candidate responders andcandidates for adverse events (resistance, recurrence, etc.) responsiveto different immunotherapies.

In various embodiments, the report generated by the process 222 mayinclude immune contexture profiling. In various examples, the process222 generates a cancer immunogram as a report for individualizedprediction of immunotherapy response. In some examples, the process 222generates integrates gene sequencing data for further analysis. Forexample, the process 222 can identify specific PD1-positivesubpopulations in NSCLC via integration of RNAseq analysis and multipleximaging. Thus, in some examples, the process 222 provides a trainedcomputational pathology machine learning model that identifies PD1 Tcells in pre-treatment biopsies from H&E stain images and correlateswith treatment response to immune checkpoints.

FIG. H illustrates an example report HOO in the form of a cancerimmunogram score that may be included in a patient report. Each spokerepresents a scale. In the illustrated example, the scales areforeignness of the tumor, the general immune status, immune cellinfiltration, absence of checkpoints, absence of soluble inhibitors,absence of inhibitory tumor metabolism, and tumor sensitivity to immuneeffectors, each further identified with italicized wording under thescale title representing an example measurement that may be representedby the scale (e.g., mutational load as a measurement of tumorforeignness). On each scale, the score associated with the biologicalspecimen may become a vertex of a polygon, for example, the polygonoutlined by a thick black line FIG. H. The area contained within thepolygon may be used as the overall immunotherapy response score (in thisexample, an immunogram score) for a biological specimen. In an example,the larger values on each scale may be closer to the perimeter of theouter circle and the smaller values may be closer to the centroid of thecircle. A larger immunogram score (i.e., larger area of the polygon) mayindicate a biological specimen that is more likely to react toimmunotherapy. Therefore, the process 222 may be configured to determineany of a plurality of scales, determine a value for each, and determinean overall immunogram score such as by determining a resulting polygonalarea. In an example, the larger values on each scale may be closer tothe centroid of the circle and the smaller values may be closer to theperimeter of the blue circle. In such examples, a smaller immunogramscore (area of the polygon), may indicate a biological specimen that ismore likely to react to immunotherapy.

In some examples, the report from process 222 associates animmunotherapy response score with a stored unmarked H&E image. In someexamples, the immunotherapy response score is based on immunotherapyassociated sequencing data, Immune Cell Infiltration, Immune GeneExpression Signatures, Multiplex PD-L1 and CD8 staining, and/orMultiplex macrophage IHC panels. If the immunotherapy response scorecontains a plurality of these different features, the immunotherapyresponse score is considered a multifaceted response score. Inparticular, multifaceted response scores may be used as scoresdetermined based on imaging features and genetic features. In any case,as exemplified in FIG. 8, the report from process 222 may be amulti-model and multi-biomarker immunogram derived from analyzingreceived H&E images and associated patient data.

Example 1

As noted above, in various embodiments, selection of the IHC stainingprotocol may include performing an optimization on the stainingprotocol. An optimization, as may be performed using the process 208 inFIG. 2, is provided for collecting ground truth annotations for H&E lungtissue models development with chromogenic multiplex IHC.

In this example, the specimen cohort included whole tissue sections ofNSCLC (adenocarcinoma) with an equal number of PDL1+ and PDL1− cases(estimated via IHC TC scoring). The cohort included only primaryresection specimens with recently-collected samples to avoid any tissuealterations that may affect molecular expressions (e.g., slide age mayalter PDL1 expression). In addition to a multiplex slide, i.e., having amultiplex IHC stain, serial sections were extracted from each tissueblock to test each antibody in a multiplex panel of antibodies/IHCstains, where the order of each tissue section was logged.

In this example, tissue slides were sectioned in the following order,with two multiplex IHC stain images:

Order Section type Markers 1 Four serial IHC sections CD3, CD8, CD20,CD68 2 H&E N/A 3 7-plex CD3, CD8, CD20, CD68, CK, PD1, PDL1 4 4-plexCD3, CD8, CD20, CD68, 5 3-plex CK, PD1, PDL1 6 Three serial IHC sectionsCK, PD1, PDL1

In particular, 11 sections were sliced from each tissue block. Thefirst, second, third, and fourth (the first four sections) were IHCstained. Each of the first four sections had one of the followingsingle-plex IHC targets such that each target is used for one section:CD3, CD8, CD20, or CD68. The fifth section was H&E stained. The sixthsection was multiplex IHC stained for seven targets: CD3, CD8, CD20,CD68, CK, PD1, PDL1. The seventh section was multiplexed IHC stained forfour targets: CD3, CD8, CD20, CD68. The eighth section was multiplexedIHC stained for three targets: CK, PD1, PDL1. Each of the ninth, tenth,and eleventh sections (the last three sections) were IHC stained. Eachof the last three sections had one of the following single-plex IHCtargets such that each target is used for one section: CK, PD1, PDL1.

Each antibody in the multiplex IHC staining panel was first optimizedindividually to define a staining reference library for final staining.In various embodiments, a staining reference is one or more tissuespecimens known to have either the presence of the stain antibody'starget molecule (positive control tissue) or the absence of the stainantibody's target molecule (negative control tissue). A stainingreference library may be commercially available. In one example, theantibody and corresponding staining reference library may be purchasedfrom or provided by the same entity.

The staining reference(s) may be used to determine whether the stain isfunctioning as expected. For example, stain should be detectable in thepositive control tissue after applying the antibody stain to thepositive control tissue. Stain should not be detectable in the negativecontrol tissue after applying the antibody stain to the negative controltissue. In one example, if the visible stain in the stained negativecontrol tissue is below an intensity level known in the art and/or ifthe stain is faintly visible throughout the tissue but not localized,the target molecule may be determined to be absent from the tissue. Ifthe staining references are not stained as expected when the antibodystain is applied to them, then the staining protocol may be adjusted(for example, optimized). Staining references may be used to compare twostaining protocols to select a protocol that produces staining that ismore accurate, easier to interpret, and/or optimal according to anothercriterion.

Ways of adjusting a staining protocol are known in the art. For example,an antibody in a stain may be replaced by another antibody having thesame target molecule. The chromogen in a stain may be replaced with adifferent chromogen. The concentration of a stain may be increased ordecreased. The amount of time that a stain is in contact with the tissuemay be changed. For multiplex staining, each stain may be applied in acertain order relative to the other stains, and the order may bechanged.

The staining reference may be used to set thresholds that will be usedto determine whether a visibly stained area in a separate tissuespecimen (for example, a specimen having an unknown presence/absencestatus of the target molecule) is determined to be an artifact(indicating the absence of target molecule in that area) or confirmedstaining indicating the present of the target molecule. In one example,the threshold is an intensity threshold and the value is higher than theintensity of any stained area in the negative control tissue and lowerthan the intensity of any stained area in the positive control tissue.

A staining reference tissue specimen may be applicable to multipleantibody stains. For example, a positive control tissue specimen mayhave more than one class of target molecules present. Similarly, anegative control tissue specimen may have more than one class of targetmolecules absent. One or more antibody stains targeting the present orabsent molecules may be applied to the tissue specimen. After the one ormore antibody stains is applied to the staining reference tissuespecimen(s), the specimen(s) may be used to set a threshold for any ofthe applied antibody stains as described above.

Positive Antibody Cell type Staining Pattern Species Clone Controls CD3T cells Cytoplasm, cell Rabbit Mono 2GV6 Tonsil membrane CD8 Cytotoxic TCell Rabbit Mono SP57 Tonsil lymphocytes membrane CD20 B Cell Mouse L26Tonsil lymphocytes membrane Mono CD68 Macrophages Cytoplasm, cell MouseKP-1 Tonsil and some membrane Mono myeloid elements PanCK Epithelialcells, Cytoplasm Mouse AE1/AE3/ Intestine, Liver, occasional Mono PCK26Skin stromal staining PD-1 T-cells, B-cells, Cytoplasm Mouse NAT105Tonsil activated Mono monocytes PD-L1 Tumor and Cytoplasm, cell RabbitMono SP142 Tonsil SP142 immune cells membrane

Further in some examples, optimization processes examine the cell type,applying a cell type rule, to identify multiplex combinations to achievedesired combinatorial effects. For example, the optimization process maycalculate for each cell, for each marker, what percentage of total stainon the cell is associated with the marker (e.g., calculating theproportions of each stained molecule associated with the individualcell). For example, if a cell has two or more classes of stainedmolecule (for example, two or more target molecules) associated with it(for example, the stained molecules appear to be located on and/or inthe cell), for each class of stained molecule, a ratio, proportion, orpercentage (for example, proportion or percentage of total stainedmolecules associated with the cell) may be calculated. In some examples,the cell type rule is configured to avoid overlapping cell type targetsacross IHC stains. In some examples, the cell type rule is configured toprovide for overlapping of at least some cell type targets. In someexamples, the cell type rule is configured to ensure that a thresholdnumber of different cell types are targeted. In some examples, the celltype rule is configured to target certain cell types associated withassociated data, such as the cancer type, patient information, tissuetype, and organ type/biopsy site. Various stainers may be used. In someexamples, all tissue sections were stained with a Ventana DISCOVERYULTRA Research autostainer. Generally, all single-plex IHC slidestaining (e.g., the first four and last three slices in this example)are stained following the manufacturer's staining protocols for eachantibody using hematoxylin as counterstain.

In some examples, brightfield image capture is used to obtain thehistology images. While brightfield image of H&E stained slices may beperformed using a DISCOVERY Ultra system using the above antibodies, forIHC stained slices, which may be Immunofluorescent (IF) slides, amodified approach may be used. For example, the specimen may be frozenif specificity/sensitivity is as good for each marker as it is FFPE,since a frozen specimen would allow more tissue and less tissuedegradation. In another example, the staining protocol process maycalculate for each cell, for each marker, what percent of total stain onthe cell is associated with the marker.

In various examples, a multiplexing IHC staining protocol optimizationprocess occurs as follows. At a first step, the staining reference foreach antibody in the final multiplex panel is defined. Positive controltissue blocks may be used to test different chromogens and antibodyorders. In other examples, chromogens and antibody orders may be storedin a dataset and accessed by the process. For example, chromogens mayinclude 3,3′-Diaminobenzidine (DAB), Red, Teal, Purple, Yellow, Blue,and Silver. One chromogen may be selected for each IHC target. Anexample antibody chromogen table is as follows:

Antibody DAB Red Teal Purple Yellow Blue Silver CD3 1 CD8 1 CD20 1 CD681 PanCK 1 PD-1 1 PD-L1 1

In the table, each row represents an antibody class (labeled by theantibody target), and each column represents a chromogen. Ideally, eachchromogen should only be conjugated to one antibody class, to make iteasier to distinguish distinct antibody classes when two or moreantibody classes are combined in an mIHC slide. A number 1 in a cellindicates that the chromogen of that column is conjugated with theantibody class of that row. Ideally, each row and each column shouldhave only one instance of the number 1. Any combination of chromogen andantibody may be used for mIHC staining, especially if the combinationsatisfies the ideal described in this paragraph.

At a second step, the sequence of chromogens used for protein detectionis determined. For example, primary antibodies from different speciesmay be cocktailed for multiplex IHC staining without intermediatedestaining, but with sequential detection to avoid any enzymaticdeposition. In other examples, primary antibodies from the same speciesmay be used sequentially with DAB counterstain to avoid falsecolocalization. Denaturing and elution steps will be applied betweeneach sequence to avoid cross-reactivity. For all co-localized markers(e.g., CD3 and CD8), the lowest expressed marker will be applied first.Incubation times for each IHC staining may follow the manufacturer'srecommendation for each antibody selected in the panel.

In an example, 50 multiplex IHC slide images (of NSCLC adeno) were usedas ground truths to train an machine learning H&E model to detectlymphocytes, macrophages, and tumor cells, and thereby provide a trainedmachine learning model to derive spatial tumor-immune profiles.

Various modalities may be used to stain and image multiplex stainslices. Virtual multi-staining may be used for whole slide imaging andmay include automatically co-registering sections, including IHC stainedand scanned serial sections. Typically, 3-way to 5-way multiplex stainslices may be formed and imaged. Multiplex chromogenic IHC provideswhole slide imaging and uses simultaneously/sequential application ofimmunostaining without the removal of previous markers. Typically, 3-wayto 5-way multiplex stain slices may be formed and imaged. Multiplexedimmunohistochemical consecutive staining on single slide (MICSSS) may beused for whole slide imaging. Typically, up to and including 10-waymultiplex stain slices may be formed and imaged. MICSSS uses iterativecycles of immunostaining, scanning, removal of chromogenic enzymesubstrate and blocking previous primary antibody. In an example, amultiplex IHC stain may be from a multiplexed chromogenic IHC assayformed using a multiplexed immunohistochemical consecutive staining onsingle slide (MICSSS) process, e.g., using markers CD8, CD68, CD3, PD1,PDL1, and PanCK. In various implementations, the multiplex IHC stainingmay comprise multiplex immunofluorescence (IF) used for whole slideimaging or region of interest imaging. Thus references herein tomultiplex IHC images include references to multiplex IF images.Multiplex IF staining includes iterative cycles of immunostaining usingtyramide signal amplification (TSA) or DNA barcodes. Typically, up toand including 5-8-way multiplex staining can be performed using TSAbased staining and up to and including 30-60-way non-TSA based, cycledstaining approaches. Example multiplex IF processes included MultiOmyxstaining or hyperplexed IF Assay, Tissue-based cyclF, and CODEX. Otherexample techniques for amplification of the epitope detection, inaddition to TSA, include nanocrystal quantum dots, Hapten-based modifiedmultiplexing. MICSSS and multiplex IF processes provide high-throughputmultiplex staining and standardized quantitative analysis that allowsfor fast and efficient ground truth training of a machine learningmodel.

FIG. 4 illustrates a system 300 for generating training data and fortraining a histology image-based machine learning model. The system 300may be used to train machine learning model trained for variousbiomarker determinations, including PD-L1 identification, immune celltype identification (for example, B cell, T cell, CD4 T cell, CD8 Tcell), whether the immune cells are located in the tumor region. Inaccordance with various embodiments herein, the machine learning modelis trained to receive histology images, e.g., H&E stained images, andperform tissue classification and cell/molecule classification, forexample. Further, the system 300 may be configured to generate animmunotherapy response class based on the number of predicted moleculesand/or locations of the predicted molecules. In some implementations, animmunotherapy response class is assigned by comparing the number ofpredicted molecules to a threshold for each molecule. In someimplementations, an immunotherapy response class is assigned bycomparing locations of predicted molecules to molecule locationcriteria. In some examples, the system 300 is configured to predictbiomarker status, tumor status, and tumor statistics, based on thetissue and cell/molecule classifications. In some implementations, theimmunotherapy response class is one of low, medium, and high lymphocyteinfiltration. For example, in some implementations, patients having lowtumor infiltration (e.g., low lymphocyte infiltration) could be athigher risk for (i.e., more likely to experience) a progression eventwhile receiving immunotherapy than patients having medium tumorinfiltration. The same may be the case for patients having medium tumorinfiltration (i.e., more likely to progress) versus patients having highinfiltration. Physicians may choose to monitor patients more frequentlyif they have a higher progression risk (for example, by ordering morefrequent imaging—CT scans, MRI, etc.—, ordering genetic testing ofeither cell-free nucleic acids or cell-associated nucleic acids, ororder other follow-up testing).

In various embodiments, in a training mode, The system 300 receivestraining data in the form of one or more unmarked digital images of ahistopathology slide, such as unmarked H&E slide images 302. In someexamples, the system 300 creates a high-density, grid-based digitaloverlay map that identifies the majority class of tissue visible withineach grid tile in the digital image. In some examples, the system 300may generate a digital overlay drawing identifying each cell in ahistopathology image, at the resolution level of an individual pixel.

The system 300 includes a machine learning model 304 that, when trained,includes a tiling and tissue detection controller 306, histology imagecell/molecule classifier 308, and a histology image tissue classifier310. In the illustrated example, each of the H&E-based classifiers areneural networks trained to classify received unmarked H&E images, forexample, take from a biopsy specimen of a patient.

To affect training of the model 304, the model 304 receives trainingdata containing unmarked H&E slide images 302 and data f14 associatedwith these unmarked H&E slide images. In various embodiments, theassociated data 312 may include marked H&E slide images 316, where suchH&E images may be automatically marked to label molecules and locationsof molecules derived from multiplex IHC stain images associated with H&Eimages, in accordance with processes described herein. For example, themarked H&E images 316 may be of slices taken from the same specimen usedto generate the slices stained to form the unmarked H&E slide images302.

In various embodiments, the associated data 312 fed to the machinelearning model 304 includes immunotherapy response score data 318associated with the H&E slide images 302, where that association isbased on clinical data. As illustrated, the immunotherapy response scoremay be of different forms and includes data such as sequencing data,Immune Cell Infiltration, Immune Gene Expression Signatures, MultiplexPD-L1 and CD8 staining, and Multiplex macrophage IHC panels. Theimmunotherapy response score may be based on a single one of thesefeatures or a combination of features resulting in a multifacetedresponse score, e.g., one based on imaging features and geneticfeatures. Multifaceted response scores may be calculated through variousmathematical operators, such as through adding individual responsescores, multiplying individual response scores, vectorising individualresponse scores, etc. and with or without feature dependent weightingfactors. Other techniques for determining response scores are describedherein and in applications incorporated by reference herein. In yetother embodiments, the associated data 312 may be sequencing data suchas tumor mutational burden (TMB), microsatellite Instability (MSI), andT Cell Clonality.

The machine learning model 304 may be part of a deep learning frameworkto execute processes described in examples herein. The machine learningmodel 304 includes a pre-processing controller 320 that performs variousimage processing features. For example, for received unmarked and markedH&E images, the controller 320 may perform aligning/registering of thereceived images, 302 and 312, such that for each physical location inthe biological specimen, pixels associated with that physical locationare aligned.

In the illustrated example, the controller 320 includes the tiling &tissue detection controller 306 that performs tiling on the receivedimages, separating each into tiles, where the tiles in any image may behave associated tiles in any other image. In this way, tile levelclassification and analysis can be performed by the model 304. That is,the cell classifier 308 and the tissue classifier 310 may each beconfigured as a tile-based neural network classifiers. Not shown, butresulting from the classifiers, the trained machine learning model mayinclude different biomarker classification models, each configured tohave a different neural network architecture and to identify a differentbiomarker in received unmarked H&E images. While in some embodiments themodel 304 is configured for training using tile training images, inother architectures the model fo04 is configured for training using notile images, but rather uses whole slide images. Example neural networkarchitecture types for the classifiers 308 and 310 include, ResNet-34,Fully convolutional network (FCN) (FIGS. 6A-6C), Inception-v3, and UNet(see, e.g., FIG. 7). Further still, in some examples, the tiling &tissue detection controller 306 may deploy a neural network trained toperforming tiling on received images and trained to perform tissuedetection within each tile.

In various embodiments, the pre-processing controller 320 may be amultiple instance learning (MIL) controller configured to separatereceived images into a plurality of tile images each corresponding to adifferent portion of the digital image, and the controller 320 usesthose tile images in training. Further, in some examples, the controller320 may use a tile selection process to select between tiles which tilesare to be used for training. In some embodiments, the pre-processingcontroller 320 is a configured in a feedback configuration that allowsfor combining a tile section process with a classification model,allowing the tile section process to be informed by a neural networkarchitecture, such as an FCN architecture. In some examples, the tileselection process performed by the controller 320 is a trained MILprocess. For example, the machine learning model 304 may generate asimulated dataset 322 during training, where that output is used as aninitial input to guide a tile selection process of controller 320.

In various embodiments, the machine learning model 304 is configuredwith a multi-tile algorithm that concurrently analyzes many tiles inimages, both individually and in conjunction with the portion of theimage that surrounds each tile. The multi-tile algorithm may achieve amultiscale, multiresolution analysis that captures both the contents ofthe individual tile and the context of the portion of the image thatsurrounds the tile. Because the portions of the image that surround twoneighboring tiles overlap, analyzing many tiles and their surroundingsconcurrently instead of separately analyzing each tile with itssurroundings reduces computational redundancy and results in greaterprocessing efficiency.

In an example, the machine learning model 304 may store the analysisresults in a 3-dimensional probability data array, which contains one1-dimensional data vector for each analyzed tile. In one example, eachdata vector contains a list of percentages that sum to 100%, eachindicating the probability that each grid tile contains one of thetissue classes analyzed. The position of each data vector in theorthogonal 2-dimensional plane of the data array, with respect to theother vectors, corresponds with the position of the tile associated withthat data vector in the digital image, with respect to the other tiles.

In the illustrated example, the pre-processing controller 320 includesthe tiling and tissue detection controller 306. In various embodiments,the controller 306 performs tissue detection executes an image tilingprocess that selects and applies a tiling mask to the received images toparse the images into small sub-images (tiles) for use in training theclassifiers 308 and 310. The controller 306 may store a plurality ofdifferent tiling masks and select a tiling mask. In some examples, theimage tiling process selects one or more tiling masks optimized fordifferent biomarkers, i.e., in some examples, image tiling is biomarkerspecific. This allows, for example, to have tiles of different pixelsizes and different pixel shapes that are selected specifically toincrease accuracy and/or to decrease processing time associated with aparticular biomarker. For example, tile sizes optimized for identifyingthe presence of TILs in an image may be different from tile sizesoptimized for identifying PD-L1 or another immunotherapy biomarker. Assuch, in some examples, the pre-processor controller 320 is configuredto perform imaging processing and tiling specific to a type ofbiomarker, and after the classifiers 308 and 310 analyze images data forthat biomarker, the controller 320 may re-process the original imagedata (e.g., unmarked and marked training H&E images) for analyzing forthe next biomarker, and so on, until all biomarkers have been examinedfor.

Generally speaking, the tiling masks applied by the image tiling processof the controller 306 may be selected to increase efficiency ofoperation of the machine learning model 304. The tiling mask may beselected based on the size of the received image data, based on theconfiguration of the machine learning model 304, or some combinationthereof.

Tiling masks may vary in the size of tiling blocks. Some tiling maskshave uniform tiling blocks, i.e., each the same size. Some tiling maskshaving tiling blocks of different sizes. The tiling mask applied by theimage tiling process may be chosen based on the number of classificationlayers in the cell classifier 308 and in the tissue classifier 310, forexample. In some examples, the tiling mask may be chosen based on theprocessor configuration of the biomarker prediction system, for example,if the multiple parallel processors are available or if graphicalprocessing units or tensor processing units are used.

In various embodiments, the cell classifier 308 may be configured as athree-class semantic segmentation FCN model developed by modifying aUNet classifier (see, e.g., FIG. 7) replacing a loss function with across-entropy function, focal loss function, or mean square errorfunction to form a three-class segmentation model. Three-class nature ofthe FCN model means that the cell classifier 308 may be configured as afirst pixel-level FCN model, that identifies and assigns each pixel ofimage data into a cell-subunit class: (i) cell interior, (ii) a cellborder, or (iii) a cell exterior. This is provided by way of example.The segmentation size of the cell classifier 308 may be determined basedon the type of cell to be segmented. For both TILs biomarkers, forexample, the cell classifier 308 may be configured to perform lymphocyteidentification and segmentation using a three-class FCN model. Forexample, the cell classifier 308 may be configured to classify pixels inan image as corresponding to the (i) interior, (ii) border, or (iii)exterior of lymphocyte cell. The cell classifier 308 may be configuredto identify and segment any number of cells, examples of which includetumor positive, tumor negative, lymphocyte positive, lymphocytenegative, immune cells, including lymphocytes, cytotoxic T cells, Bcells, NK cells, macrophages, etc.

The use of a three-class model facilitates, among other things, thecounting of each individual cell, especially when two or more cellsoverlap each other for more accurate classification. Tumor infiltratinglymphocytes will overlap tumor cells. In traditional two-class celloutlining models that only label whether a pixel contains a cell outeredge or not, each clump of two or more overlapping cells would becounted as one cell, which can produce inaccurate results.

In addition to using a three-class model, the cell classifier 308 may beconfigured to avoid the possibility that a cell that spans two tiles iscounted twice, by adding a buffer around all four sides of each tilethat is slightly wider than an average cell. The intention is to onlycount cells that appear in the center, non-buffered region for eachtile. In this case, tiles will be placed so that the center,non-buffered region of neighboring tiles are adjacent andnon-overlapping. Neighboring tiles will overlap in their respectivebuffer regions.

In one example, the cell segmentation algorithm of the classifier 308may be formed of two UNet models. One UNet model may be trained withimages of mixed tissue classes, where a human analyst has highlightedthe outer edge of each cell and classified each cell according to tissueclass. In one example, training data includes digital slide images whereevery pixel has been labeled as either the interior of a cell, the outeredge of a cell, or the background which is exterior to every cell. Inanother example, the training data includes digital slide images whereevery pixel has been labeled with a yes or no to indicate whether itdepicts the outer edge of a cell. This UNet model can recognize theouter edges of many types of cells and may classify each cell accordingto cell shape or its location within a tissue class region assigned bythe tissue classifier 310.

Another UNet model may be trained with images of many cells of a singletissue class, or images of a diverse set of cells where cells of onlyone tissue class are outlined in a binary mask. In one example, thetraining set is labeled by associating a first value with all pixelsshowing a cell type of interest and a second value to all other pixels.Visually, an image labeled in this way might appear as a black and whiteimage wherein all pixels showing a tissue class of interest would bewhite and all other pixels would be black, or vice versa. For example,the images may have only labeled lymphocytes. This UNet model canrecognize the outer edges of that particular cell type and assign alabel to cells of that type in the digital image of the slide.

The tissue classifier 310 may be configured to classify tissue in a tileas corresponding to one of a number of tissue classes, such as biomarkerstatus, tumor status, tissue type, and/or tumor state/condition, orother information. In an example implementation of a TILs biomarker (anexample immunotherapy biomarker), the tissue classifier 310 may classifytissue using tissue classifications, such as Tumor—IHC positive,Tumor—IHC negative, Necrosis, Stroma, Epithelium, or Blood. The tissueclassifier 310 may identify boundaries for the different tissue typesand generates metadata for use in visually display boundaries and colorcoding for different tissue types in an overlay mapping reportgenerator.

In various embodiments, the tissue classifier 310 is a tile-basedclassifier configured to classify tiles as corresponding to one of aplurality of different tissue classifications. Examples of tissueclasses include but are not limited to tumor, stroma, normal,lymphocyte, fat, muscle, blood vessel, immune cluster, necrosis,hyperplasia/dysplasia, red blood cells, and tissue classes or cell typesthat are positive (contain a target molecule of an IHC stain, especiallyin a quantity larger than a certain threshold) or negative for an IHCstain target molecule (do not contain that molecule or contain aquantity of that molecule lower than a certain threshold). Examples alsoinclude tumor positive, tumor negative, lymphocyte positive, andlymphocyte negative.

In various embodiments, the tissue classifier 310 is configured as aneural network having an FCN architecture, an example of which isillustrated in FIGS. 6A-6C, and discussed further below.

In various embodiments, the cell classifier 308 is configured as aneural network having a UNet architecture, an example of which isillustrated in FIG. 7.

In FIG. 5, example system 400 is provided for generating training dataand for training a histology image-based machine learning model, as anexample implementation of the system 300 of FIG. 4. Multiplex IHC images402 are received and provided to a multiplex IHC deconvolver 404. Insome examples, an IHC staining process 202 may be used to generate theimages 402. In some examples, the images 402 are received from anexternal image source. For each received multiplex IHC image, thedeconvolver 404 detects mixture colors containing more than one IHCstain and identifies the IHC stains that comprise each mixture color, byusing a deconvolving process, such as described in 216 of FIG. 2. Forexample, the deconvolver 404 may determine the location of each IHCstain color by setting an intensity threshold for each stain color andcomparing the intensity of the stain color in each pixel to theintensity threshold for that stain color. In some examples, thedeconvolver 404 generates an overlay for each IHC stain where each pixelhaving an intensity that exceeds the threshold for the IHC stain isannotated to indicate presence of the IHC stain in the pixel.

In some examples, the deconvolver 404 generates a plurality of IHC stainspecific images for each multiplex IHC image, where those specificimages correspond to a different IHC stain. The deconvolver 404 mayadditionally generate an overlap stain image(s) comprising overlappingIHC stain regions. The output images and image data from the deconvolveris provided to an IHC machine learning framework 406 having a stainedtarget identifier 408 and a cell location identifier 410.

The stained target identifier 408 is configured to examine received IHCimages, including single-plex and multiplex IHC images from thedeconvolver 404 and determine the location of the associated stainedtarget molecules, targeted by one or more IHC stains forming themultiplex IHC image. Target molecules may be identified in each IHCslide image. In an example implementation, the target molecule in afirst tissue layer IHC image is CD3; the target molecule in a secondtissue layer IHC image is CD8; the target molecule in a third tissuelayer IHC image is CD20; the target molecule in a fourth tissue layerIHC image is CD68; the target molecules in a sixth tissue layermultiplex IHC image are CD3, CD8, CD20, CD68, CK, PD1, and PDL1; thetarget molecules in a seventh tissue layer multiplex IHC image are CD3,CD8, CD20, and CD68; the target molecules in an eighth tissue layermultiplex IHC image are CK, PD1, and PDL1; the target molecule in aninth tissue layer IHC image is CK; the target molecule in a tenthtissue layer IHC image is PD1; and the target molecule in an eleventhtissue layer IHC image is PDL1.

In the illustrated example, the cell location identifier 410 is a neuralnetwork trained to detect individual cell locations and determines celltype (i.e., inferred cell type). For example, the cell locationidentifier 410 may detect which individual cells are lymphocytes in themultiplex IHC image(s) and determine their locations. The cell locationidentifier 410 may be configured in a UNET neural network, such ashaving the configuration like that illustrated in FIG. 7.

The machine learning framework 406 annotates the IHC images, single-plexand multiplex images, with the identified target molecules andidentified cell locations and cell types (i.e., inferred cell types) andprovides these as IHC ground truth mask images 412. For training themachine learning model 404, these ground truth masks 412 are provided toan H&E slide labeler 414. The labeler 414 performs alignment and/orregistration of the ground truth masks 412 to received unmarked H&Eslide images 416, for example, such that for each physical location inthe biological specimen, all pixels associated with that physicallocation are aligned. The labeler 414 marks the location on the unmarkedH&E image that corresponds to the locations of the target moleculesstained on the IHC ground truth masks 412, thereby generating one ormore marked H&E slide images 418. In various embodiments, for each cellin the H&E image having a location that corresponds to the location ofone of the IHC ground truth mask images 412, the labeler 414 maycalculate the percentage of stained pixels overlapping the cell that isassociated with each IHC stain to determine an IHC stain profile foreach cell and store the same as image data of the marked H& slide image.

In marked H&E slide images 418 and the unmarked H&E slide images 416 areprovided to the machine learning model 304 for training the classifiers308 and 310, in accordance with techniques described herein.

FIG. 8 illustrates example images of H&E images at different stages ofprocessing, in an example implementation. An H&E image 700 has beenseparated into tiles and each tile classified by a tissue classificationprocess, identifying the tile as corresponding to one of plurality ofdifferent tissue types: epithelium, immune, stroma, tumor, or other. Theclassified tiles may then be each provided to a cell classifier. Or, insome examples, only certain classified tiles are provided for cellclassification, such as tiles classified as corresponding to tumortissue. Multiple instance learning models may be used in a tileclassifier to extract only tumor tiles, for example. Image 702 is anexample tile after a cell classifier has identified nuclei within thetile and performed a nuclei segmentation highlight the nuclei. Image 704is an example tile after the cell classifier has performed the cellsegmentation and then performs a cell detection to identify, in thisexample, lymphocyte cells and their locations in the tile. This processis repeated across all tiles forming the H&E image 700 or across asubset of the tiles thereof, such as the tumor classified tiles. Fromthere, the machine learning model may derive various statistics. Examplestatistics include a “TIL score” for the image, that is, from count ofall lymphocyte cells detected within tumor tiles (see, FIG. 9).Estimated tumor percentage, that is, a ratio of all tiles classified astumor over total tissue tiles, is another statistic. Other additionalstatistics include whole image-level cell counts (mean, standarddeviation, quartiles, etc.). Cell morphology features may be determined,such as texture-based features derived from the detected cells. Fractalsand tumor nuclei morphology may be determined using Delaunaytriangulation (w/o normalization) techniques, for example. These andother statistics may be calculated by a tumor report generator, such asreport generator 180, or other processor or process. In implementations,various statistics can be calculated from the output of the trainedmachine learning models, including one or more of percentage of cellshaving a particular molecule, percentage of cells having a particularratio of molecules, location relationships among cell types, extent ofmixing of cell types, and degree of tumor infiltration by lymphocytes.Such statistics may be determined from classification data, inparticular, from the number of predicted molecules and locations of thepredicted molecules contained in the classification data. In someimplementations, these statistics are determined by a machine learningmodel, such as machine learning model 304. In any event, any of suchdata may be provided to a pathologist in an automatically generatedreport. From these labeled H&E images and developed statistics, H&Ederived features may be determined, such as assigning an immunotherapyresponse class. Examples include predicting immunotherapy responseoutcome for patients identified with NSCLC, predicting colorectal cancerand lung metastasis. These may be presented to a pathologist in agenerated report.

FIG. 6A illustrates an example neural network architecture 500 that maybe used for tissue classification, such as that performed by tissueclassifier 310 in FIG. 4 and/or process 216 in FIG. 2. In particular,the architecture 500 has a tile-resolution FCN configuration. As shown,the tile-resolution FCN configuration included in the tissue classifier310 has additional layers of 1×1 convolution in a skip connection,downsampling by a factor of 8 in a skip connection, and a confidence maplayer, and replaces an average pooling layer with a concatenation layer,and a fully connected FCN layer with a 1×1 convolution and Softmaxlayer. The added layers convert a classification task into aclassification-segmentation task. This means that instead of receivingand classifying a whole image as one tissue class label, the addedlayers allow the tile-resolution FCN to classify each small tile in theuser-defined grid as a tissue class.

These added and replacement layers convert a CNN to a tile-resolutionFCN without requiring the upsampling performed in the later layers oftraditional pixel-resolution FCNs. Upsampling is a method by which a newversion of an original image can be created with a higher resolutionvalue than the original image. However, upsampling is a time-consuming,computation-intense process, which can be avoided with the presentarchitecture. FIG. 6B illustrates example output sizes for differentlayers and resulting sub-layers of the architecture 500.

There are many methods known in the art for upsampling, includingnearest-neighbor, bilinear, hermite, bell, Mitchell, bicubic, andLanczos resampling. In one example, 2× upsampling means that a pixelwith red green blue (RGB) values will be split into four pixels, and theRGB values for the three new pixels may be selected to match the RGBvalues of the original pixel. In another example, the RGB values for thethree new pixels may be selected as the average of the RGB values fromthe original pixel and the pixels that are adjacent to the neighboringpixel.

Because the RGB values of the new pixels may not accurately reflect thevisible tissue in the original slide that was captured by the digitalslide image, upsampling can introduce errors into the final image.

In an example, instead of labeling individual pixels, thetile-resolution FCN architecture 500 is programmed to analyze a largesquare tile made of small square tiles, producing a 3D array of valuesthat each represent the probability that one tissue class classificationlabel matches the tissue class depicted in each small tile. Aconvolution layer performs the multiplication of at least one inputimage matrix by at least one filter matrix. In the first convolutionlater, the input image matrix has a value for every pixel in the largesquare tile input image, representing visual data in that pixel (forexample, a value between 0 and 255 for each channel of RGB).

The filter matrix may have dimensions selected by the user, and maycontain weight values selected by the user or determined bybackpropagation during CNN model training. In one example, in the firstconvolution layer, the filter matrix dimensions are 7×7 and there are 64filters. The filter matrix may represent visual patterns that candistinguish one tissue class from another.

In an example where RGB values populate the input image matrix, theinput image matrix and the filter matrices will be 3-dimensional. Eachfilter matrix is multiplied by each input image matrix to produce aresult matrix. All result matrices produced by the filters in oneconvolution layer may be stacked to create a 3-dimensional result matrixhaving dimensions such as rows, columns, and depth. The last dimension,depth, in the 3-D result matrix will have a depth equal to the number offilter matrices. The resulting matrix from one convolution layer becomesthe input image matrix for the next convolution layer.

A convolution layer title that includes “/n”, where n is a number,indicates that there is a downsampling (also known as pooling) of theresult matrix produced by that layer. The n indicates the factor bywhich the downsampling occurs. Downsampling by a factor of 2 means thata downsampled result matrix with half as many rows and half as manycolumns as the original result matrix will be created by replacing asquare of four values in the result matrix by one of those values or astatistic calculated from those values. For example, the minimum,maximum, or average of the values may replace the original values.

The architecture 500 also adds skip connections (shown in FIG. 6A asblack lines with arrows that connect convolution layers directly to theconcatenation layer). The skip connection on the left includesdownsampling by a factor of 8, and the skip connection on the rightincludes two convolution layers that multiply an input image matrix byfilter matrices that each have dimensions of 1×1. Because of the 1×1dimensions of the filter matrices in these layers, only an individualsmall square tile contributes to its corresponding probability vector inthe result matrices created by the purple convolution layers. Theseresult matrices represent a small focus of view.

In all of the other convolution layers, the larger dimensions of thefilter matrices allow the pixels in each medium square tile, includingthe small square tile at the center of the medium square tile, tocontribute to the probability vector in the result matrix thatcorresponds with that small square tile. These result matrices allow thecontextual pixel data patterns surrounding the small square tile toinfluence the probability that each tissue class label applies to thesmall square tile. These result matrices represent a large focus ofview.

The 1×1 convolution layers in the skip connection allow the algorithm toregard the pixel data patterns in the center small square tile as eithermore or less important than pixel data patterns in the rest of thesurrounding medium square tile. The amount of importance is reflected bythe weights that the trained model multiplies by the final result matrixfrom the skip connection layers (shown on the right side of FIG. 6A)compared to the weights that the trained model multiplies by the finalresult matrix from the medium tile convolution layers during theconcatenation layer.

The downsampling skip connection shown on the left side of FIG. 6Acreates a result matrix with a depth of 64. The 3×3 convolution layerhaving 512 filter matrices creates a result matrix with a depth of 512.The 1×1 convolution layer having 64 filter matrices creates a resultmatrix with a depth of 64. All three of these results matrices will havethe same number of rows and the same number of columns. Theconcatenation layer concatenates these three results matrices to form afinal result matrix with the same number of rows and the same number ofcolumns as the three concatenated matrices, and a depth of 64+512+64(640). This final result matrix combines the large and small focus ofview matrices.

The final result matrix may be flattened to 2 dimensions by multiplyinga factor by every entry, and summing the products along each depth. Eachfactor may be selected by the user, or may be selected during modeltraining by backpropagation. Flattening will not change the number ofrows and columns of the final results matrix, but will change the depthto 1.

The 1×1 convolution layer receives the final result matrix and filtersit with one or more filter matrices. The 1×1 convolution layer mayinclude one filter matrix associated with each tissue class label in thetrained algorithm. This convolution layer produces a 3-D result matrixthat has a depth equal to the number of tissue class labels. Each depthcorresponds to one filter matrix and along the depth of the resultmatrix there may be a probabilities vector for each small square tile.This 3-D result matrix is the 3-dimensional probability data array, andthe 1×1 convolution layer stores this 3-D probability data array.

A Softmax layer may create a 2-dimensional probability matrix from the3-D probability data array by comparing every value in eachprobabilities vector and selecting the tissue class associated with themaximum value to assign that tissue class to the small square tileassociated with that probabilities vector.

The stored 3-dimensional probability data array or the 2-D probabilitymatrix may then be converted to a tissue class overlay map in the finalconfidence map layer, to efficiently assign a tissue class label to eachtile.

In one example, to counteract shrinkage, input image matrices have addedrows and columns on all four outer edges of the matrices, wherein eachvalue entry in the added rows and columns is a zero. These rows andcolumns are referred to as padding. In this case, the training datainput matrices will have the same number of added rows and columns withvalue entries equal to zero. A difference in the number of padding rowsor columns in the training data input matrices would result in values inthe filter matrices that do not cause the tissue class locator 216 toaccurately label input images.

In the FCN shown in FIG. 6A, 217 total outer rows or columns on eachside of the input image matrix will be lost to shrinkage before the skipconnection, due to the gray and blue layers. Only the pixels located inthe small square tiles will have a corresponding vector in the resultmatrices created by the green layers and beyond.

In one example, each medium square tile is not padded by adding rows andcolumns with value entries of zero around the input image matrix thatcorresponds to each medium square tile because the zeroes would replaceimage data values from neighboring medium square tiles that the tissueclass locator 216 needs to analyze. In this case, the training datainput matrices will not be padded either.

FIG. 6C is a visualization of each depth of an exemplary 3-dimensionalinput image matrix being convoluted by two exemplary 3-dimensionalfilter matrices. In an example where an input image matrix contains RGBchannels for each medium square tile, the input image matrix and filtermatrices will be 3-dimensional. In one of the three dimensions, theinput image matrix and each filter matrix will have three depths, onefor red channel, one for green channel, and one for blue channel.

The red channel (first depth) 502 of the input image matrix ismultiplied by the corresponding first depth of the first filter matrix.The green channel (second depth) 504 is multiplied in a similar fashion,and so on with the blue channel (third depth) 506. Then, the red, green,and blue product matrices are summed to create a first depth of the3-dimensional result matrix. This repeats for each filter matrix, tocreate an additional depth of the 3-dimensional result matrix thatcorresponds to each filter.

As shown in FIG. 10, the output from the trained machine learning model304 may be provided to a immunotherapy response decision system 800(such as may be part of a genomic sequencing system, oncology system,chemotherapy decision system, immunotherapy decision system, or othertherapy decision system) that determines a listing of possibleimmunotherapies based on received data, i.e., the H&E image(s) withtissue and cell/molecule classifications biomarker reports, such asstatistics including those based on the biomarker metrics, genomicsequencing data, etc. and other statistics determined by the machinelearning model 304. The system 800 analyzes the classificationinformation and or statistics and other received molecular data againstavailable immunotherapies 802, and the system 800 recommends a matchedlisting of possible tumor-type specific immunotherapies 804, filteredfrom the list of available immunotherapies 802, in the form of a matchedtherapy report. In some examples, the listing of immunotherapies isranked in an order as determined by the decision system 800, forexample, a ranked order based on predicted likelihood of success amongthe matched listing of possible immunotherapies.

In various examples, the techniques herein may be deployed partially orwholly within a dedicated slide imager, such as a high throughputdigital scanner. FIG. 11 illustrates an example system 900 having adedicated ultrafast pathology (slide) scanner system 902, such as aPhilips IntelliSite Pathology Solution available from KoninklijkePhilips N.V. of Amsterdam, Netherlands. In some examples, the pathologyscanner system 902 may contain a plurality of trained biomarkerclassification models. Exemplary models may include, for instance, thosedisclosed in U.S. application Ser. No. 16/412,362. The scanner system902 is coupled to an imaging-based biomarker prediction system 904,implementing processes as discussed and illustrated in examples herein.For example, in the illustrated example, the system 904 includes a deeplearning framework 906 based on tile-based classification modules, inaccordance with examples herein, having multiplex IHC image ground truthmask generating neural network 908, a trained cell classifier 910 and atrained tissue classifier 912. Through the NN 908, the deep learningframework 906 generates ground truth masks from multiplex IHC imageswhere those masks are used to train the cell classifier 910 and tissueclassifiers 912. The deep learning framework 906 performs biomarker andtumor classifications on histopathology images, e.g., unmarked H&E slideimages, and stores the classification data as overlay data with theoriginal images in a generated images database 914. The images may besaved as TIFF files, for example. Although the database 914 may includeJSON files and other data generated by the classification processesherein. In some examples, the deep learning framework may be integratedin whole or in part within the scanner 902, as shown in optional block915.

To manage generated images, which can be quite large, an imagemanagement system and viewer generator 916 is provided. In theillustrated example, the system 916 is illustrated as external to theimaging-based biomarker prediction system 904, connected by a private orpublic network. Yet, in other examples, all or part of the system 916may be deployed in the system 904, as shown at 919. In some examples,the system 916 is cloud based, and stores generated images from (orinstead of) the database 0914. In some examples, the system 3016generates a web-accessible cloud based viewer, allowing pathologists toaccess, view, and manipulate, through a graphic user interface,histopathology images with various classification overlays.

In some examples, the image management system 916 manages receipt ofscanned slide images 918 from the scanner 902, where these slide imagesare generated from an imager 920.

In the illustrated example, the image management system 916 generates anexecutable viewer App 924 and deploys that App 924 to an App DeploymentEngine 922 of the scanner 902. The App Deployment Engine 922 may providefunctionality such as GUI generation allowing users to interact with theview App 924, an App marketplace allowing users to download the viewerApp 924 from the image management system 916 or from other networkaccessible sources.

FIG. 12 illustrates an example computing device 1000 for implementingthe imaging-based biomarker prediction system 100 of FIG. 1. Asillustrated, the system 100 may be implemented on the computing device1000 and in particular on one or more processing units 1010, which mayrepresent Central Processing Units (CPUs), and/or on one or more orGraphical Processing Units (GPUs) 1011, including clusters of CPUsand/or GPUs, and/or one or more tensor processing unites (TPU) (alsolabeled 1011), any of which may be cloud based. Features and functionsdescribed for the system 100 may be stored on and implemented from oneor more non-transitory computer-readable media 1012 of the computingdevice 1000. The computer-readable media 1012 may include, for example,an operating system 1014 and the deep learning framework 1016 havingelements corresponding to that of machine learning model 304, deeplearning framework 906 and deep learning framework 915, including thepre-processing controllers and classifies therein. More generally, thecomputer-readable media 1012 may store trained deep learning models,executable code, etc. used for implementing the techniques herein. Thecomputer-readable media 1012 and the processing units 1010 andTPU(S)/GPU(S) 1011 may store histology images and data, tissueclassification data, cell segmentation data, lymphocyte segmentationdata, TILs metrics, and other data herein in one or more databases 1013.The computing device 1000 includes a network interface 1024communicatively coupled to the network 1050, for communicating to and/orfrom a portable personal computer, smart phone, electronic document,tablet, and/or desktop personal computer, or other computing devices.The computing device further includes an I/O interface 1026 connected todevices, such as digital displays 1028, user input devices 1030, etc. Insome examples, as described herein, the computing device 1000 generatesbiomarker prediction as an electronic document 1015 that can be accessedand/or shared on the network 1050. In the illustrated example, thesystem 100 is implemented on a single server 1000. However, thefunctions of the system 100 may be implemented across distributeddevices 1000, 1002, 1004, etc. connected to one another through acommunication link. In other examples, functionality of the system 100may be distributed across any number of devices, including the portablepersonal computer, smart phone, electronic document, tablet, and desktoppersonal computer devices shown. In other examples, the functions of thesystem 100 may be cloud based, such as, for example one or moreconnected cloud TPU (s) customized to perform machine learningprocesses. The network 1050 may be a public network such as theInternet, private network such as research institution's orcorporation's private network, or any combination thereof. Networks caninclude, local area network (LAN), wide area network (WAN), cellular,satellite, or other network infrastructure, whether wireless or wired.The network can utilize communications protocols, including packet-basedand/or datagram-based protocols such as internet protocol (IP),transmission control protocol (TCP), user datagram protocol (UDP), orother types of protocols. Moreover, the network can include a number ofdevices that facilitate network communications and/or form a hardwarebasis for the networks, such as switches, routers, gateways, accesspoints (such as a wireless access point as shown), firewalls, basestations, repeaters, backbone devices, etc.

The computer-readable media may include executable computer-readablecode stored thereon for programming a computer (e.g., comprising aprocessor(s) and GPU(s)) to the techniques herein. Examples of suchcomputer-readable storage media include a hard disk, a CD-ROM, digitalversatile disks (DVDs), an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. More generally, the processing units of the computing device1300 may represent a CPU-type processing unit, a GPU-type processingunit, a TPU-type processing unit, a field-programmable gate array(FPGA), another class of digital signal processor (DSP), or otherhardware logic components that can be driven by a CPU.

It is noted that while example deep learning frameworks and neuralnetworks herein have been described as configured with example machinelearning architectures (FCN configurations and UNET configurations), anynumber of suitable convolutional neural network architectures may beused. Broadly speaking, the deep learning frameworks herein mayimplement any suitable statistical model (e.g., a neural network orother model implemented through a machine learning process) that will beapplied to each of the received images. As discussed herein, thatstatistical model may be implemented in a variety of manners. In someexamples, machine learning is used to evaluate training images anddevelop classifiers that correlate predetermined image features tospecific categories of TILs status. In some examples, image features canbe identified as training classifiers using a learning algorithm such asNeural Network, Support Vector Machine (SVM) or other machine learningprocess. Once classifiers within the statistical model are adequatelytrained with a series of training images, the statistical model may beemployed in real time to analyze subsequent images provided as input tothe statistical model for predicting biomarker status. In some examples,when a statistical model is implemented using a neural network, theneural network may be configured in a variety of ways. In some examples,the neural network may be a deep neural network and/or a convolutionalneural network. In some examples, the neural network can be adistributed and scalable neural network. The neural network may becustomized in a variety of manners, including providing a specific toplayer such as but not limited to a logistics regression top layer. Aconvolutional neural network can be considered as a neural network thatcontains sets of nodes with tied parameters. A deep convolutional neuralnetwork can be considered as having a stacked structure with a pluralityof layers. The neural network or other machine learning processes mayinclude many different sizes, numbers of layers and levels ofconnectedness. Some layers can correspond to stacked convolutionallayers (optionally followed by contrast normalization and max-pooling)followed by one or more fully-connected layers. For neural networkstrained by large datasets, the number of layers and layer size can beincreased by using dropout to address the potential problem ofoverfitting. In some instances, a neural network can be designed toforego the use of fully connected upper layers at the top of thenetwork. By forcing the network to go through dimensionality reductionin middle layers, a neural network model can be designed that is quitedeep, while dramatically reducing the number of learned parameters.

Thus, as provided, a system for performing the methods described hereinmay include a computing device, and more particularly may be implementedon one or more processing units, for example, Central Processing Units(CPUs), and/or on one or more or Graphical Processing Units (GPUs),including clusters of CPUs and/or GPUs. Features and functions describedmay be stored on and implemented from one or more non-transitorycomputer-readable media of the computing device. The computer-readablemedia may include, for example, an operating system and softwaremodules, or “engines,” that implement the methods described herein. Moregenerally, the computer-readable media may store batch normalizationprocess instructions for the engines for implementing the techniquesherein. The computing device may be a distributed computing system, suchas an Amazon Web Services cloud computing solution.

The functions of the engines may be implemented across distributedcomputing devices, etc. connected to one another through a communicationlink. In other examples, functionality of the system may be distributedacross any number of devices, including the portable personal computer,smart phone, electronic document, tablet, and desktop personal computerdevices shown. The computing device may be communicatively coupled tothe network and another network. The networks may be public networkssuch as the Internet, a private network such as that of a researchinstitution or a corporation, or any combination thereof. Networks caninclude, local area network (LAN), wide area network (WAN), cellular,satellite, or other network infrastructure, whether wireless or wired.The networks can utilize communications protocols, includingpacket-based and/or datagram-based protocols such as Internet protocol(IP), transmission control protocol (TCP), user datagram protocol (UDP),or other types of protocols. Moreover, the networks can include a numberof devices that facilitate network communications and/or form a hardwarebasis for the networks, such as switches, routers, gateways, accesspoints (such as a wireless access point as shown), firewalls, basestations, repeaters, backbone devices, etc.

The methods and systems described above may be utilized in combinationwith or as part of a digital and laboratory health care platform that isgenerally targeted to medical care and research. It should be understoodthat many uses of the methods and systems described above, incombination with such a platform, are possible. One example of such aplatform is described in U.S. Patent Publication No. 2021/0090694,titled “Data Based Cancer Research and Treatment Systems and Methods”,and published Mar. 25, 2021, which is incorporated herein by referenceand in its entirety for any and all purposes.

For example, an implementation of one or more embodiments of the methodsand systems as described above may include microservices constituting adigital and laboratory health care platform supporting immunotherapyresponse prediction from H&E images. Embodiments may include a singlemicroservice for executing and delivering immunotherapy responseprediction or may include a plurality of microservices each having aparticular role which together implement one or more of the embodimentsabove. In one example, a first microservice may execute H&E imageanalysis and molecule location prediction in order to deliver predictedmolecule locations to a second microservice for analyzing number andlocation of predicted molecules. Similarly, the second microservice mayexecute analysis of predicted molecule locations to deliverimmunotherapy response prediction according to an embodiment, above.

Where embodiments above are executed in one or more micro-services withor as part of a digital and laboratory health care platform, one or moreof such micro-services may be part of an order management system thatorchestrates the sequence of events as needed at the appropriate timeand in the appropriate order necessary to instantiate embodiments above.A micro-services based order management system is disclosed, forexample, in U.S. Patent Publication No. 2020/80365232, titled “AdaptiveOrder Fulfillment and Tracking Methods and Systems”, and published Nov.19, 2020, which is incorporated herein by reference and in its entiretyfor all purposes.

For example, continuing with the above first and second microservices,an order management system may notify the first microservice that anorder for H&E image analysis and molecule location prediction has beenreceived and is ready for processing. The first microservice may executeand notify the order management system once the delivery of predictedmolecule locations is ready for the second microservice. Furthermore,the order management system may identify that execution parameters(prerequisites) for the second microservice are satisfied, includingthat the first microservice has completed, and notify the secondmicroservice that it may continue processing the order to analyzepredicted molecule locations and generate immunotherapy responseprediction according to an embodiment, above.

Where the digital and laboratory health care platform further includes agenetic analyzer system, the genetic analyzer system may includetargeted panels and/or sequencing probes. An example of a targeted panelis disclosed, for example, in U.S. Patent Publication No. 2021/0090694,titled “Data Based Cancer Research and Treatment Systems and Methods”,and published March 25, 2021, which is incorporated herein by referenceand in its entirety for all purposes. An example of a targeted panel forsequencing cell-free (cf) DNA and determining various characteristics ofa specimen based on the sequencing is disclosed, for example, in U.S.patent application Ser. No. 17/179,086, titled “Methods And Systems ForDynamic Variant Thresholding In A Liquid Biopsy Assay”, and filed Feb.18, 2021, U.S. patent application Ser. No. 17/179,267, titled“Estimation Of Circulating Tumor Fraction Using Off-Target Reads OfTargeted-Panel Sequencing”, and filed Feb. 18, 2021, and U.S. patentapplication Ser. No. 17/179,279, titled “Methods And Systems ForRefining Copy Number Variation In A Liquid Biopsy Assay”, and filed Feb.18, 2021 which is incorporated herein by reference and in its entiretyfor all purposes. In one example, targeted panels may enable thedelivery of next generation sequencing results (including sequencing ofDNA and/or RNA from solid or cell-free specimens) according to anembodiment, above. An example of the design of next-generationsequencing probes is disclosed, for example, in U.S. Patent PublicationNo. 2021/0115511, titled “Systems and Methods for Next GenerationSequencing Uniform Probe Design”, and published Jun. 22, 2021 and U.S.patent application Ser. No. 17/323,986, titled “Systems and Methods forNext Generation Sequencing Uniform Probe Design”, and filed May 18,2021, which are incorporated herein by reference and in their entiretyfor all purposes.

Where the digital and laboratory health care platform further includesan epigenetic analyzer system, the epigenetic analyzer system mayanalyze specimens to determine their epigenetic characteristics and mayfurther use that information for monitoring a patient over time. Anexample of an epigenetic analyzer system is disclosed, for example, inU.S. patent application Ser. No. 17/352,231, titled “Molecular ResponseAnd Progression Detection From Circulating Cell Free DNA”, and filedJun. 18, 2021, which is incorporated herein by reference and in itsentirety for all purposes.

Where the digital and laboratory health care platform further includes abioinformatics pipeline, the methods and systems described above may beutilized after completion or substantial completion of the systems andmethods utilized in the bioinformatics pipeline. As one example, thebioinformatics pipeline may receive next-generation genetic sequencingresults and return a set of binary files, such as one or more BAM files,reflecting DNA and/or RNA read counts aligned to a reference genome.

When the digital and laboratory health care platform further includes anRNA data normalizer, any RNA read counts may be normalized beforeprocessing embodiments as described above. An example of an RNA datanormalizer is disclosed, for example, in U.S. Patent Publication No.2020/0098448, titled “Methods of Normalizing and Correcting RNAExpression Data”, and published Mar. 26, 2020, which is incorporatedherein by reference and in its entirety for all purposes.

When the digital and laboratory health care platform further includes agenetic data deconvolver, any system and method for deconvolving may beutilized for analyzing genetic data associated with a specimen havingtwo or more biological components to determine the contribution of eachcomponent to the genetic data and/or determine what genetic data wouldbe associated with any component of the specimen if it were purified. Anexample of a genetic data deconvolver is disclosed, for example, in U.S.Patent Publication No. 2020/0210852, published Jul. 2, 2020, andPCT/US19/69161, filed Dec. 31, 2019, both titled “TranscriptomeDeconvolution of Metastatic Tissue Samples”; and U.S. patent applicationSer. No. 17/074,984, titled “Calculating Cell-type RNA Profiles forDiagnosis and Treatment”, and filed Oct. 20, 2020, the contents of eachof which are incorporated herein by reference and in their entirety forall purposes.

RNA expression levels may be adjusted to be expressed as a valuerelative to a reference expression level. Furthermore, multiple RNAexpression data sets may be adjusted, prepared, and/or combined foranalysis and may be adjusted to avoid artifacts caused when the datasets have differences because they have not been generated by using thesame methods, equipment, and/or reagents. An example of RNA data setadjustment, preparation, and/or combination is disclosed, for example,in U.S. patent application Ser. No. 17/405,025, titled “Systems andMethods for Homogenization of Disparate Datasets”, and filed Aug. 18,2021.

When the digital and laboratory health care platform further includes anautomated RNA expression caller, RNA expression levels associated withmultiple samples may be compared to determine whether an artifact iscausing anomalies in the data. An example of an automated RNA expressioncaller is disclosed, for example, in U.S. Pat. No. 11,043,283, titled“Systems and Methods for Automating RNA Expression Calls in a CancerPrediction Pipeline”, and issued Jun. 22, 2021, which is incorporatedherein by reference and in its entirety for all purposes.

The digital and laboratory health care platform may further include oneor more insight engines to deliver information, characteristics, ordeterminations related to a disease state that may be based on geneticand/or clinical data associated with a patient, specimen and/ororganoid. Exemplary insight engines may include a tumor of unknownorigin (tumor origin) engine, a human leukocyte antigen (HLA) loss ofhomozygosity (LOH) engine, a tumor mutational burden engine, a PD-L1status engine, a homologous recombination deficiency engine, a cellularpathway activation report engine, an immune infiltration engine, amicrosatellite instability engine, a pathogen infection status engine, aT cell receptor or B cell receptor profiling engine, a line of therapyengine, a metastatic prediction engine, an IO progression riskprediction engine, and so forth.

An example tumor origin or tumor of unknown origin engine is disclosed,for example, in U.S. patent application Ser. No. 15/930,234, titled“Systems and Methods for Multi-Label Cancer Classification”, and filedMay 12, 2020, which is incorporated herein by reference and in itsentirety for all purposes.

An example of an HLA LOH engine is disclosed, for example, in U.S. Pat.No. 11,081,210, titled “Detection of Human Leukocyte Antigen Class ILoss of Heterozygosity in Solid Tumor Types by NGS DNA Sequencing”, andissued Aug. 3, 2021, which is incorporated herein by reference and inits entirety for all purposes. An additional example of an HLA LOHengine is disclosed, for example, in U.S. patent application Ser. No.17/304,940, titled “Detection of Human Leukocyte Antigen Loss ofHeterozygosity”, and filed Jun. 28, 2021, which is incorporated hereinby reference and in its entirety for all purposes.

An example of a tumor mutational burden (TMB) engine is disclosed, forexample, in U.S. Patent Publication No. 2020/0258601, titled“Targeted-Panel Tumor Mutational Burden Calculation Systems andMethods”, and published Aug. 13, 2020, which is incorporated herein byreference and in its entirety for all purposes.

An example of a PD-L1 status engine is disclosed, for example, in U.S.Patent Publication No. 2020/0395097, titled “A Pan-Cancer Model toPredict The PD-L1 Status of a Cancer Cell Sample Using RNA ExpressionData and Other Patient Data”, and published Dec. 17, 2020, which isincorporated herein by reference and in its entirety for all purposes.An additional example of a PD-L1 status engine is disclosed, forexample, in U.S. Pat. No. 10,957,041, titled “Determining Biomarkersfrom Histopathology Slide Images”, issued Mar. 23, 2021, which isincorporated herein by reference and in its entirety for all purposes.

An example of a homologous recombination deficiency engine is disclosed,for example, in U.S. Pat. No. 10,975,445, titled “An IntegrativeMachine-Learning Framework to Predict Homologous RecombinationDeficiency”, and issued Apr. 13, 2021, which is incorporated herein byreference and in its entirety for all purposes. An additional example ofa homologous recombination deficiency engine is disclosed, for example,in U.S. patent application Ser. No. 17/492,518, titled “Systems andMethods for Predicting Homologous Recombination Deficiency Status of aSpecimen”, filed Oct. 1, 2021, which is incorporated herein by referenceand in its entirety for all purposes.

An example of a cellular pathway activation report engine is disclosed,for example, in U.S. Patent Publication No. 2021/0057042, titled“Systems And Methods For Detecting Cellular Pathway Dysregulation InCancer Specimens”, and published Feb. 25, 2021, which is incorporatedherein by reference and in its entirety for all purposes.

An example of an immune infiltration engine is disclosed, for example,in U.S. Patent Publication No. 2020/0075169, titled “A Multi-ModalApproach to Predicting Immune Infiltration Based on Integrated RNAExpression and Imaging Features”, and published Mar. 5, 2020, which isincorporated herein by reference and in its entirety for all purposes.

An example of an MSI engine is disclosed, for example, in U.S. PatentPublication No. 2020/0118644, titled “Microsatellite InstabilityDetermination System and Related Methods”, and published Apr. 16, 2020,which is incorporated herein by reference and in its entirety for allpurposes. An additional example of an MSI engine is disclosed, forexample, in U.S. Patent Publication No. 2021/0098078, titled “Systemsand Methods for Detecting Microsatellite Instability of a Cancer Using aLiquid Biopsy”, and published Apr. 1, 2021, which is incorporated hereinby reference and in its entirety for all purposes.

An example of a pathogen infection status engine is disclosed, forexample, in U.S. Pat. No. 11,043,304, titled “Systems And Methods ForUsing Sequencing Data For Pathogen Detection”, and issued Jun. 22, 2021,which is incorporated herein by reference and in its entirety for allpurposes. Another example of a pathogen infection status engine isdisclosed, for example, in PCT/US21/18619, titled “Systems And MethodsFor Detecting Viral DNA From Sequencing”, and filed Feb. 18, 2021, whichis incorporated herein by reference and in its entirety for allpurposes.

An example of a T cell receptor or B cell receptor profiling engine isdisclosed, for example, in U.S. patent application Ser. No. 17/302,030,titled “TCR/BCR Profiling Using Enrichment with Pools of CaptureProbes”, and filed Apr. 21, 2021, which is incorporated herein byreference and in its entirety for all purposes.

An example of a line of therapy engine is disclosed, for example, inU.S. Patent Publication No. 2021/0057071, titled “Unsupervised LearningAnd Prediction Of Lines Of Therapy From High-Dimensional LongitudinalMedications Data”, and published Feb. 25, 2021, which is incorporatedherein by reference and in its entirety for all purposes.

An example of a metastatic prediction engine is disclosed, for example,in U.S. Pat. No. 11,145,416, titled “Predicting likelihood and site ofmetastasis from patient records”, and issued Oct. 12, 2021, which isincorporated herein by reference and in its entirety for all purposes.

An example of an IO progression risk prediction engine is disclosed, forexample, in U.S. patent application Ser. No. 17/455,876, titled“Determination of Cytotoxic Gene Signature and Associated Systems andMethods For Response Prediction and Treatment”, and filed Nov. 19, 2021,which is incorporated herein by reference and in its entirety for allpurposes.

An additional example of a microsatellite instability engine isdisclosed, for example, in U.S. patent application Ser. No. 16/412,362,titled “A Generalizable and Interpretable Deep Learning Framework forPredicting MSI From Histopathology Slide Images”, and filed May 4, 2019,which is incorporated herein by reference and in its entirety for allpurposes.

An example of a radiomics engine is disclosed, for example, in U.S.patent application Ser. No. 16/460,975, titled “3D Radiomic Platform forImaging Biomarker Development”, and filed Jul. 2, 2019, which isincorporated herein by reference and in its entirety for all purposes.

An example of a tissue segmentation engine is disclosed, for example, inU.S. patent application Ser. No. 16/732,242, titled “ArtificialIntelligence Segmentation Of Tissue Images”, and filed Dec. 31, 2019,which is incorporated herein by reference and in its entirety for allpurposes.

When the digital and laboratory health care platform further includes areport generation engine, the methods and systems described above may beutilized to create a summary report of a patient's genetic profile andthe results of one or more insight engines for presentation to aphysician. For instance, the report may provide to the physicianinformation about the extent to which the specimen that was sequencedcontained tumor or normal tissue from a first organ, a second organ, athird organ, and so forth. For example, the report may provide a geneticprofile for each of the tissue types, tumors, or organs in the specimen.The genetic profile may represent genetic sequences present in thetissue type, tumor, or organ and may include variants, expressionlevels, information about gene products, or other information that couldbe derived from genetic analysis of a tissue, tumor, or organ.

The report may include therapies and/or clinical trials matched based ona portion or all of the genetic profile or insight engine findings andsummaries. For example, the clinical trials may be matched according tothe systems and methods disclosed in U.S. Patent Publication No.2020/0381087, titled “Systems and Methods of Clinical Trial Evaluation”,published Dec. 3, 2020, which is incorporated herein by reference and inits entirety for all purposes.

The report may include a comparison of the results (for example,molecular and/or clinical patient data) to a database of results frommany specimens. An example of methods and systems for comparing resultsto a database of results are disclosed in U.S. Patent Publication No.2020/0135303 titled “User Interface, System, And Method For CohortAnalysis” and published Apr. 30, 2020, and U.S. Patent Publication No.2020/0211716 titled “A Method and Process for Predicting and AnalyzingPatient Cohort Response, Progression and Survival”, and published Jul.2, 2020, which is incorporated herein by reference and in its entiretyfor all purposes. The information may be used, sometimes in conjunctionwith similar information from additional specimens and/or clinicalresponse information, to match therapies likely to be successful intreating a patient, discover biomarkers or design a clinical trial.

Any data generated by the systems and methods and/or the digital andlaboratory health care platform may be downloaded by the user. In oneexample, the data may be downloaded as a CSV file comprising clinicaland/or molecular data associated with tests, data structuring, and/orother services ordered by the user. In various embodiments, this may beaccomplished by aggregating clinical data in a system backend, andmaking it available via a portal. This data may include not onlyvariants and RNA expression data, but also data associated withimmunotherapy markers such as MSI and TMB, as well as RNA fusions.

When the digital and laboratory health care platform further includes adevice comprising a microphone and speaker for receiving audible queriesor instructions from a user and delivering answers or other information,the methods and systems described above may be utilized to add data to adatabase the device can access. An example of such a device isdisclosed, for example, in U.S. Patent Publication No. 2020/0335102,titled “Collaborative Artificial Intelligence Method And System”, andpublished Oct. 22, 2020, which is incorporated herein by reference andin its entirety for all purposes.

When the digital and laboratory health care platform further includes amobile application for ingesting patient records, including genomicsequencing records and/or results even if they were not generated by thesame digital and laboratory health care platform, the methods andsystems described above may be utilized to receive ingested patientrecords. An example of such a mobile application is disclosed, forexample, in U.S. Pat. No. 10,395,772, titled “Mobile Supplementation,Extraction, And Analysis Of Health Records”, and issued Aug. 27, 2019,which is incorporated herein by reference and in its entirety for allpurposes. Another example of such a mobile application is disclosed, forexample, in U.S. Pat. No. 10,902,952, titled “Mobile Supplementation,Extraction, And Analysis Of Health Records”, and issued Jan. 26, 2021,which is incorporated herein by reference and in its entirety for allpurposes. Another example of such a mobile application is disclosed, forexample, in U.S. Patent Publication No. 2021/0151192, titled “MobileSupplementation, Extraction, And Analysis Of Health Records”, and filedMay 20, 2021, which is incorporated herein by reference and in itsentirety for all purposes.

When the digital and laboratory health care platform further includesorganoids developed in connection with the platform (for example, fromthe patient specimen), the methods and systems may be used to furtherevaluate genetic sequencing data derived from an organoid and/or theorganoid sensitivity, especially to therapies matched based on a portionor all of the information determined by the systems and methods,including predicted cancer type(s), likely tumor origin(s), etc. Thesetherapies may be tested on the organoid, derivatives of that organoid,and/or similar organoids to determine an organoid's sensitivity to thosetherapies. Any of the results may be included in a report. If theorganoid is associated with a patient specimen, any of the results maybe included in a report associated with that patient and/or delivered tothe patient or patient's physician or clinician. In various examples,organoids may be cultured and tested according to the systems andmethods disclosed in U.S. Patent Publication No. 2021/0155989, titled“Tumor Organoid Culture Compositions, Systems, and Methods”, publishedMay 27, 2021; PCT/US20/56930, titled “Systems and Methods for PredictingTherapeutic Sensitivity”, filed Oct. 22, 2020; U.S. Patent PublicationNo. 2021/0172931, titled “Large Scale Organoid Analysis”, published Jun.10, 2021; PCT/US2020/063619, titled “Systems and Methods for HighThroughput Drug Screening”, filed Dec. 7, 2020 and U.S. patentapplication Ser. No. 17/301,975, titled “Artificial Fluorescent ImageSystems and Methods”, filed Apr. 20, 2021 which are each incorporatedherein by reference and in their entirety for all purposes. In oneexample, the drug sensitivity assays may be especially informative ifthe systems and methods return results that match with a variety oftherapies, or multiple results (for example, multiple equally orsimilarly likely cancer types or tumor origins), each matching with atleast one therapy.

When the digital and laboratory health care platform further includesapplication of one or more of the above in combination with or as partof a medical device or a laboratory developed test that is generallytargeted to medical care and research, such laboratory developed test ormedical device results may be enhanced and personalized through the useof artificial intelligence. An example of laboratory developed tests,especially those that may be enhanced by artificial intelligence, isdisclosed, for example, in U.S. Patent Publication No. 2021/0118559,titled “Artificial Intelligence Assisted Precision Medicine Enhancementsto Standardized Laboratory Diagnostic Testing”, and published Apr. 22,2021, which is incorporated herein by reference and in its entirety forall purposes.

It should be understood that the examples given above are illustrativeand do not limit the uses of the systems and methods described herein incombination with a digital and laboratory health care platform.

Additional Aspects

Aspect 1: A method for using a machine learning model to analyze atleast one hematoxylin and eosin (H&E) slide image, the methodcomprising: a. receiving, at one or more processors, the H&E slideimage; b. using, at the one or more processors, a machine learning modelto predict locations of molecules in the H&E slide image, where themachine learning model is trained using a training data set comprising aplurality of unmarked H&E images and a plurality of marked H&E images,each marked H&E image being associated with one unmarked H&E image andeach marked H&E image including a location of one or more moleculesdetermined by analyzing a multiplex IHC image having at least two IHCstains, wherein each IHC stain has a unique color and a unique targetmolecule and wherein analyzing the multiplex IHC image includesdetermining an IHC stain that contributes to any two or more overlappingor adjacent IHC stains and comparing each IHC stain in the multiplex IHCimage to a threshold; c. analyzing the number of predicted molecules andlocations of the predicted molecules; and d. assigning an immunotherapyresponse class to the H&E slide image, based on the number of predictedmolecules and/or locations of the predicted molecules.

Aspect 2: The method of Aspect 1 where the molecules are immunotherapybiomarkers selected from the group consisting of CD3, CD8, CD20, CD68,CK, PD1, and PDL1.

Aspect 3: The method of Aspect 1, further comprising locating individualcells.

Aspect 4: The method of Aspect 3, further comprising inferring, usingthe machine learning model, cell types for at least one of theindividual cells.

Aspect 5: The method of Aspect 4, further comprising: predicting animmunotherapy response of the patient based, at least partially, on theinferred cell types.

Aspect 6: The method of Aspect 3, further comprising, for eachindividual cell associated with two or more classes of stainedmolecules, calculating the proportions of each stained moleculeassociated with the individual cell.

Aspect 7: The method of Aspect 6, further comprising: predicting animmunotherapy response of the patient based, at least partially, on thecalculated proportions of each stained molecule associated with eachindividual cell.

Aspect 8: The method of Aspect 1, further comprising calculating amultifaceted score based on imaging features and genetic features.

Aspect 9: The method of Aspect 1, further comprising calculatingadditional statistics from the number of predicted molecules andlocations of the predicted molecules.

Aspect 10: The method of Aspect 9, where the additional statisticsinclude at least one of: percentage of cells having a particularmolecule, percentage of cells having a particular ratio of molecules,location relationships among cell types, extent of mixing of cell types,and degree of tumor infiltration by lymphocytes.

Aspect 11: The method of Aspect 10, further comprising: predicting animmunotherapy response of the patient based, at least partially, on theadditional statistics.

Aspect 12: The method of Aspect 1, wherein the assigning theimmunotherapy response class includes comparing the number of predictedmolecules to a threshold for each molecule.

Aspect 13: The method of Aspect 1, where the assigning the immunotherapyresponse class includes comparing locations of predicted molecules tomolecule location criteria.

Aspect 14: The method of Aspect 1, where the immunotherapy responseclass is one of low, medium, and high lymphocyte infiltration.

Aspect 15: The method of Aspect 1, where the H&E image is associatedwith a patient.

Aspect 16: The method of Aspect 1, further comprising predicting animmunotherapy response of the patient, based on the number of predictedmolecules and locations of the predicted molecules and matching withimmunotherapy treatment.

Aspect 17: A method for using a machine learning model to analyze atleast one H&E slide image associated with a patient, comprising: a.scoring H&E slide image for similarity to slide images associated withimmunotherapy responders versus slide images associated withimmunotherapy non-responders; and b. comparing the score to a threshold.

Aspect 18: The method of Aspect 17, where the H&E image is associatedwith a tumor organoid.

Aspect 19: The method of Aspect 18, which further includes the step ofpredicting an immunotherapy response of the tumor organoid, based on thenumber of predicted molecules and locations of the predicted moleculesand predicting drug sensitivity response.

Aspect 20: A method for using a machine learning model to analyze atleast one hematoxylin and eosin (H&E) slide image associated with atumor organoid, the method comprising: a. scoring a H&E slide image forsimilarity to slide images associated with immunotherapy respondersversus slide images associated with immunotherapy non-responders; and b.comparing the score of the H&E slide image to a threshold.

Aspect 21: A method for generating training data for a histologyimage-based machine learning model, the method comprising:

-   -   a. obtaining at least one H&E slide image associated with a        biological specimen;    -   b. obtaining one or more multiplex immunohistochemistry (IHC)        images associated with the biological specimen, wherein each        multiplex IHC image includes at least two IHC stains, where each        IHC stain has a unique color and a unique target molecule;    -   c. for each multiplex IHC image, detecting mixture colors        comprised of more than one IHC stain and identifying the IHC        stains that comprise each mixture color;    -   d. determining the location of each IHC stain color and        determining the location of the associated stained target        molecules;    -   e. detecting individual cell locations and determining which        individual cells are lymphocytes;    -   f. for each H&E image and IHC image associated with the        biological specimen, align/register images such that for each        physical location in the biological specimen, all pixels        associated with that physical location are aligned;    -   g. for each target molecule, marking the location on the H&E        image that corresponds to the locations of the target molecules        stained on the IHC layers;    -   h. for each cell having a location that corresponds to the        location of one or more IHC stains, calculating the percentage        of stained pixels overlapping the cell that is associated with        each IHC stain to determine an IHC stain profile for each cell;        and    -   i. storing marked and unmarked versions of the H&E image as part        of a training data set.

Aspect 22: The method of Aspect 21, where the H&E image is captured froma tissue layer that is stained only with H&E.

Aspect 23: The method of Aspect 21, where the H&E image is captured froma tissue layer that is stained with H&E and at least one IHC stain.

Aspect 24: The method of Aspect 21, where the H&E image is a virtual H&Estain image generated based on cell and tissue structures visible in abrightfield image of a tissue layer.

Aspect 25: The method of Aspect 21, where determining the location ofeach IHC stain color includes setting an intensity threshold for eachstain color and comparing the intensity of the stain color in each pixelto the intensity threshold for that stain color.

Aspect 26: The method of Aspect 25, further comprising generating anoverlay for each IHC stain where each pixel having an intensity thatexceeds the threshold for the IHC stain is annotated to indicatepresence of the IHC stain in the pixel.

Aspect 27: The method of Aspect 21, wherein detecting cell locations isperformed by a neural network.

Aspect 28: The method of Aspect 21, where detecting cell locationsincludes the use of UNET.

Aspect 29: The method of Aspect 21, where identifying the IHC stainsthat comprise each mixture color is accomplished by deconvolving mixturecolors within each image.

Aspect 30: The method of Aspect 21, further comprising assigning atissue class to portions of the H&E image.

Aspect 31: The method of Aspect 21, further comprising associating animmunotherapy response score with the stored unmarked H&E image, basedon clinical data associated with the biological specimen.

Aspect 32: The method of Aspect 31, where the immunotherapy responsescore is based on immunotherapy associated sequencing data, Immune CellInfiltration, Immune Gene Expression Signatures, Multiplex PD-L1 and CD8staining, and Multiplex macrophage IHC panels.

Aspect 33: The method of Aspect 31, where immunotherapy associatedsequencing data includes tumor mutational burden (TMB), microsatelliteInstability (MSI), and T Cell Clonality.

Aspect 34: The method of Aspect 21, repeating (a)-(i) for a plurality ofbiological specimens to generate the training data set.

Aspect 35: The method of Aspect 1, further comprising: e. receiving thebiological specimen; f. dividing the biological specimen into aplurality of tissue layers; g. simultaneously adding at least twoclasses of antibody-conjugated (IHC) stain to one of the tissue layers,wherein each class of antibody-conjugated stain binds to a unique classof target molecule and each class of antibody-conjugated stain has aunique stain color, such that each stain color is associated with atarget molecule; and h. for each of the stained layers, capturing andstoring one digital image.

Aspect 36: The method of Aspect 35, wherein the target molecule in afirst tissue layer is CD3; the target molecule in a second tissue layeris CD8; the target molecule in a third tissue layer is CD20; the targetmolecule in a fourth tissue layer is CD68; a fifth tissue layer isstained with H&E; the target molecules in a sixth tissue layer are CD3,CD8, CD20, CD68, CK, PD1, and PDL1; the target molecules in a seventhtissue layer are CD3, CD8, CD20, and CD68; the target molecules in aneighth tissue layer are CK, PD1, and PDL1; the target molecule in aninth tissue layer is CK; the target molecule in a tenth tissue layer isPD1; and the target molecule in an eleventh tissue layer is PDL1.

Aspect 37: The method of Aspect 35, comprising simultaneously adding toone tissue layer of the tissue layers, a plurality of IHC stains suchthat the target molecules in the one tissue layer are CD3, CD8, CD20,CD68, CK, PD1, and PDL1.

Aspect 38: A method for training a histology image-based machinelearning model, the method comprising: a. receiving a training data setcomprising unmarked H&E images and data associated with each unmarkedH&E image; and b. optimizing the histology image-based machine learningmodel to receive an unmarked H&E image and generate a simulated data setsimilar to the data associated with that unmarked H&E image.

Aspect 39: The method of Aspect 38, where the associated data includes acorresponding marked H&E image for each unmarked H&E image, wherein themarked H&E image shows the location of IHC staining target molecules inone or more IHC images associated with the same biological specimen asthe H&E image, where at least one of the IHC images is a multiplex IHCimage having two or more IHC stains.

Aspect 40: The method of Aspect 38, where the associated data includesan immunotherapy response score.

Aspect 41: The method of Aspect 38, wherein the training data setcomprises a plurality of unmarked H&E images and a plurality of markedH&E images, each marked H&E image being associated with one unmarked H&Eimage and each marked H&E image including a location of one or moremolecules determined by analyzing a multiplex IHC image having at leasttwo IHC stains, wherein each IHC stain has a unique color and a uniquetarget molecule and wherein analyzing the multiplex IHC image includesdetermining an IHC stain that contributes to any two or more overlappingor adjacent IHC stains and comparing each IHC stain in the multiplex IHCimage to a threshold.

Aspect 42: The method of Aspect 38, further comprising receiving thetraining data set of Aspect 42.

Aspect 43: The method of Aspect 38, where the histology image-basedmachine learning model is a neural network.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components or multiple components.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (e.g., code embodiedon a machine-readable medium or in a transmission signal) or hardware.In hardware, the routines, etc., are tangible units capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a microcontroller, fieldprogrammable gate array (FPGA) or an application-specific integratedcircuit (ASIC)) to perform certain operations. A hardware module mayalso comprise programmable logic or circuitry (e.g., as encompassedwithin a general-purpose processor or other programmable processor) thatis temporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connects the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of the example methods described herein can beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method can be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but also deployed across a numberof machines. In some example embodiments, the processor or processorsmay be located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but also deployed across a number of machines. In some exampleembodiments, the one or more processors or processor-implemented modulesmay be located in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as an example only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

1. A method for using a machine learning model to analyze at least onehematoxylin and eosin (H&E) slide image, the method comprising: a.receiving, at one or more processors, the H&E slide image; b. using, atthe one or more processors, a machine learning model to predictlocations of molecules in the H&E slide image, where the machinelearning model is trained using a training data set comprising aplurality of unmarked H&E images and a plurality of marked H&E images,each marked H&E image being associated with one unmarked H&E image andeach marked H&E image including a location of one or more moleculesdetermined by analyzing a multiplex IHC image having at least two IHCstains, wherein each IHC stain has a unique color and a unique targetmolecule and wherein analyzing the multiplex IHC image includesdetermining an IHC stain that contributes to any two or more overlappingor adjacent IHC stains and comparing each IHC stain in the multiplex IHCimage to a threshold; c. analyzing the number of predicted molecules andlocations of the predicted molecules; and d. assigning an immunotherapyresponse class to the H&E slide image, based on the number of predictedmolecules and/or locations of the predicted molecules.
 2. The method ofclaim 1 where the molecules are immunotherapy biomarkers selected fromthe group consisting of CD3, CD8, CD20, CD68, CK, PD1, and PDL1.
 3. Themethod of claim 1, further comprising locating individual cells.
 4. Themethod of claim 3, further comprising inferring, using the machinelearning model, cell types for at least one of the individual cells. 5.The method of claim 4, further comprising: predicting an immunotherapyresponse of the patient based, at least partially, on the inferred celltypes.
 6. The method of claim 3, further comprising, for each individualcell associated with two or more classes of stained molecules,calculating the proportions of each stained molecule associated with theindividual cell.
 7. The method of claim 6, further comprising:predicting an immunotherapy response of the patient based, at leastpartially, on the calculated proportions of each stained moleculeassociated with each individual cell.
 8. The method of claim 1, furthercomprising calculating a multifaceted score based on imaging featuresand genetic features.
 9. The method of claim 1, further comprisingcalculating additional statistics from the number of predicted moleculesand locations of the predicted molecules.
 10. The method of claim 9,where the additional statistics include at least one of: percentage ofcells having a particular molecule, percentage of cells having aparticular ratio of molecules, location relationships among cell types,extent of mixing of cell types, and degree of tumor infiltration bylymphocytes.
 11. The method of claim 10, further comprising: predictingan immunotherapy response of the patient based, at least partially, onthe additional statistics.
 12. The method of claim 1, wherein theassigning the immunotherapy response class includes comparing the numberof predicted molecules to a threshold for each molecule.
 13. The methodof claim 1, where the assigning the immunotherapy response classincludes comparing locations of predicted molecules to molecule locationcriteria.
 14. The method of claim 1, where the immunotherapy responseclass is one of low, medium, and high lymphocyte infiltration.
 15. Themethod of claim 1, where the H&E image is associated with a patient. 16.The method of claim 1, further comprising predicting an immunotherapyresponse of the patient, based on the number of predicted molecules andlocations of the predicted molecules and matching with immunotherapytreatment.
 17. A method for using a machine learning model to analyze atleast one H&E slide image associated with a patient, comprising: a.scoring H&E slide image for similarity to slide images associated withimmunotherapy responders versus slide images associated withimmunotherapy non-responders; and b. comparing the score to a threshold.18. The method of claim 17, where the H&E image is associated with atumor organoid.
 19. The method of claim 18, which further includes thestep of predicting an immunotherapy response of the tumor organoid,based on the number of predicted molecules and locations of thepredicted molecules and predicting drug sensitivity response.
 20. Amethod for using a machine learning model to analyze at least onehematoxylin and eosin (H&E) slide image associated with a tumororganoid, the method comprising: a. scoring a H&E slide image forsimilarity to slide images associated with immunotherapy respondersversus slide images associated with immunotherapy non-responders; and b.comparing the score of the H&E slide image to a threshold. 21.(canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled)
 25. (canceled)26. (canceled)
 27. (canceled)
 28. (canceled)
 29. (canceled) 30.(canceled)
 31. (canceled)
 32. (canceled)
 33. (canceled)
 34. (canceled)35. The method of claim 1, further comprising: e. receiving thebiological specimen; f. dividing the biological specimen into aplurality of tissue layers; g. simultaneously adding at least twoclasses of antibody-conjugated (IHC) stain to one of the tissue layers,wherein each class of antibody-conjugated stain binds to a unique classof target molecule and each class of antibody-conjugated stain has aunique stain color, such that each stain color is associated with atarget molecule; and h. for each of the stained layers, capturing andstoring one digital image.
 36. The method of claim 35, wherein thetarget molecule in a first tissue layer is CD3; the target molecule in asecond tissue layer is CD8; the target molecule in a third tissue layeris CD20; the target molecule in a fourth tissue layer is CD68; a fifthtissue layer is stained with H&E; the target molecules in a sixth tissuelayer are CD3, CD8, CD20, CD68, CK, PD1, and PDL1; the target moleculesin a seventh tissue layer are CD3, CD8, CD20, and CD68; the targetmolecules in an eighth tissue layer are CK, PD1, and PDL1; the targetmolecule in a ninth tissue layer is CK; the target molecule in a tenthtissue layer is PD1; and the target molecule in an eleventh tissue layeris PDL1.
 37. The method of claim 35, comprising simultaneously adding toone tissue layer of the tissue layers, a plurality of IHC stains suchthat the target molecules in the one tissue layer are CD3, CD8, CD20,CD68, CK, PD1, and PDL1.
 38. A method for training a histologyimage-based machine learning model, the method comprising: a. receivinga training data set comprising unmarked H&E images and data associatedwith each unmarked H&E image; and b. optimizing the histologyimage-based machine learning model to receive an unmarked H&E image andgenerate a simulated data set similar to the data associated with thatunmarked H&E image.
 39. The method of claim 38, where the associateddata includes a corresponding marked H&E image for each unmarked H&Eimage, wherein the marked H&E image shows the location of IHC stainingtarget molecules in one or more IHC images associated with the samebiological specimen as the H&E image, where at least one of the IHCimages is a multiplex IHC image having two or more IHC stains.
 40. Themethod of claim 38, where the associated data includes an immunotherapyresponse score.
 41. The method of claim 38, wherein the training dataset comprises a plurality of unmarked H&E images and a plurality ofmarked H&E images, each marked H&E image being associated with oneunmarked H&E image and each marked H&E image including a location of oneor more molecules determined by analyzing a multiplex IHC image havingat least two IHC stains, wherein each IHC stain has a unique color and aunique target molecule and wherein analyzing the multiplex IHC imageincludes determining an IHC stain that contributes to any two or moreoverlapping or adjacent IHC stains and comparing each IHC stain in themultiplex IHC image to a threshold.
 42. The method of claim 38, furthercomprising receiving the training data set of claim
 42. 43. The methodof claim 38, where the histology image-based machine learning model is aneural network.